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Executive  Summary 


The  purpose  of  this  research  effort  was  to  select  “best  bet”  predictor  measures  that  could  be  used  to 
identify  entry-level  or  early  career  officers  and  airmen  for  Remotely-Piloted  Aircraft  (RPA)  pilot 
(officer)  or  sensor  operator  (SO;  enlisted)  positions.  When  this  project  began  in  September  2009,  a 
great  deal  of  work  had  already  been  conducted  on  RPA  operator  job  requirements  and  training. 
There  were  also  several  ongoing  initiatives  focusing  on  RPA  pilot  and  sensor  operator  (SO) 
selection,  including  efforts  conducted  by  the  71 1  HPW/HP  and  USAF  School  of  Aerospace 
Medicine  (USAFSAM/FEC),  both  at  Brooks  City  Base,  TX,  and  efforts  at 
711  HPW/RHAS  at  Mesa,  AZ.  The  Air  Force  Personnel  Center,  Research  and  Analysis  Branch 
contracted  with  the  Human  Resources  Research  Organization  (HumRRO)  to  review  and 
qualitatively  analyze  information  available  from  these  earlier  and  ongoing  efforts,  and  to  map  this 
infonnation  against  existing  taxonomies  of  SAOCs  to  detennine  if  there  are  any  gaps  in  coverage. 
We  compiled  information  about  the  skills,  abilities,  and  other  characteristics  (SAOCs)  required  in 
the  RPA  pilot  and  SO  jobs  and  used  the  U.S.  Department  of  Labor’s  0*NET  content  model  as  an 
organizing  structure  to  ensure  that  we  covered  all  of  the  important  domains  (e.g.,  abilities,  skills, 
work  style  preferences,  work  context)  and  to  provide  a  well-researched  taxonomic  structure  and 
labels  for  the  disparate  sources  of  infonnation.  RPA  subject  matter  experts  (SMEs)  and  the  Air 
Force  Personnel  Center  (AFPC)  project  team  helped  narrow  the  list  to  those  SAOCs  most  critical 
to  measure  in  an  entry-  level  selection  process.  Next,  we  worked  with  the  AFPC  team  to  take 
account  of  practical  considerations  in  the  entry-level  selection  process,  such  as  constraints  on 
testing  time  and  requirements  for  large-scale  testing.  Ultimately,  we  identified  a  “best  bet” 
predictor  and  at  least  one  alternate  predictor  for  as  many  of  the  critical  SAOCs  as  possible,  and 
then  created  two  options  for  a  recommended  entry-level  selection  battery  for  each  position  (Pilot 
and  SO).  One  option  makes  maximal  use  of  existing  operational  selection  tools  and  administration 
processes;  the  second  replaces  some  of  the  existing  selection  tools  with  others  that  have 
demonstrated  evidence  of  reliability  and  validity,  as  well  as  at  least  some  evidence  that  they  result 
in  smaller  gender  subgroup  score  differences  (and  possibly  smaller  race/ethnic  subgroup  score 
differences). 

We  also  developed  two  new  predictor  measures  to  address  measurement  gaps.  The  first  is  a 
measure  of  time  sharing  ability  that  involves  performing  multiple  tasks  simultaneously,  but 
does  not  pair  cognitive  processing  tasks  with  psychomotor  tasks.  This  measure  can  be 
administered  on  the  existing  Test  of  Basic  Aviation  Skills  (TBAS)  platfonn  and  has  a  modular 
programming  foundation  that  provides  a  great  deal  of  flexibility  for  future  research  and 
adaptations.  The  second  measure  is  an  RPA-specific  Person-Environment  (P-E)  fit  inventory.  This 
instrument  is  best  suited  for  use  as  a  self-assessment  tool  that  can  help  potential  recruits  detennine 
if  the  RPA  work  context  would  be  a  good  fit  for  their  work  preferences.  It  could  provide  an 
opening  for  recruiters  to  discuss  the  RPA  career  field  with  candidates  prior  to  accessioning. 

In  the  future,  the  USAF  can  consider  developing  measures  to  address  other  measurement  gaps 
identified  in  this  project  including  judgment  and  decision  making,  critical  thinking,  teamwork  skills, 
and/or  oral  expression  and  comprehension  skills.  Prior  to  developing  such  measures,  we  recommend 
further  exploration  of  the  likely  increment  in  prediction  beyond  that  provided  by  existing  measures, 
relative  to  the  cost  of  increased  testing  time  or  adding  a  requirement  for  non-standard  equipment 
(e.g.,  the  capability  to  capture  audio  input). 
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Identifying  Best  Bet  Entry-Level  Selection  Measures  for  US  Air  Force  Remotely 
Piloted  Aircraft  (RPA)  Pilot  and  Sensor  Operator  (SO)  Occupations 

Introduction 

The  demand  for  RPA  support  has  grown  dramatically  over  the  last  several  years  (Schanz, 
2010),  to  the  point  where  the  training  pipeline  for  RPA  pilots  and  sensor  operators  (SOs)  cannot 
meet  the  demand  ( UAS  Operator  Way  Ahead,  2008;  U.S.  GAO,  2010).  The  Air  National  Guard 
(ANG)  has  played  a  key  role  in  meeting  this  demand,  operating  RPA  squadrons  in  Arizona, 
California,  North  Dakota,  and  Texas  (Schanz,  2010).  In  addition,  the  Active  Duty  Air  Force  has 
pursued  several  strategies  for  increasing  the  number  of  RPA  pilots  and  SOs  (Church,  2011;  Tirpak, 
2010).  The  first  strategy  was  to  cross-train  rated  personnel  to  fly  RPAs.  Obviously,  these 
individuals  already  had  extensive  training  and  experience  as  a  pilot.  The  Active  Duty  Air  Force 
also  began  tunneling  some  Specialized  Undergraduate  Pilot  Training  (SUPT)  graduates  into  RPA 
training,  rather  than  into  other  air  frames.  Both  of  these  strategies  have  the  disadvantage  of 
reducing  the  number  of  rated  personnel  available  for  other  air  frames.  In  2009,  the  Air  Education 
and  Training  Command  (AETC)  beta  tested  a  program  designed  to  train  non-pilots  to  fly  RPAs. 
These  officers  included  rated  navigators  and  officers  from  non-flying  career  fields.  In  October 
2009,  the  Air  Force  created  a  dedicated  RPA  career  field,  18X,  as  a  rated  Air  Force  specialty  code 
(AFSC),  and  developed  Undergraduate  RPA  Training  (URT)  for  RPA  pilots,  with  the  first  class 
beginning  in  the  fall  of  2010  (Lyle,  2010).  The  U.S.  Air  Force  Academy  (USAFA)  also  provides 
three  introductory  courses  related  to  RPA  flight  and  management,  with  the  goal  of  feeding 
graduates  directly  into  URT  (Hoffman,  2010). 

In  addition  to  the  shortage  of  RPA  Pilots,  there  is  also  an  unmet  demand  for  RPA  SOs 
(Hoffman,  2009).  The  SO  job  is  filled  by  enlisted  personnel.  Originally,  Active  Duty  RPA  SOs 
were  drawn  from  airmen  with  a  background  in  imagery  analysis  (IA).  In  contrast,  in  the  ANG, 
SOs  were  drawn  from  a  wide  variety  of  technical  backgrounds.  In  January  2009,  the  Air  Force 
established  a  new,  dedicated  AFSC,  1U0X1,  Unmanned  Aerospace  System  (UAS)  Sensor 
Operator,  and  developed  an  SO-specific  training  course  later  that  year.  The  first  SO  training 
course  to  include  enlisted  personnel  directly  from  basic  military  training  (BMT)  occurred  in 
January  2010  (Bowlin,  2010). 


Objectives 

The  purpose  of  this  research  effort  was  to  select  “best  bet”  predictor  measures  that  could 
help  the  U.S.  Air  Force  (USAF)  identify  entry-level  or  early  career  officers  and  airmen  likely  to 
succeed  in  RPA  positions.  The  first  step  in  this  process  involved  specifying  the  job  requirements 
of  RPA  pilots  and  SOs,  specifically,  the  skills,  abilities,  and  other  characteristics  (SAOCs) 
predictive  of  success  in  RPA  pilot  or  SO  training  and  the  propensity  to  make  RPAs  a  career.  The 
SAOC  requirements,  in  turn,  provide  the  basis  for  selection  requirements  and  predictor  measures 
that  can  be  used  in  the  entry-level  selection  process.  The  USAF  has  extensive  experience  in  the 
development  and  validation  of  selection  methods  for  other  aircrew  occupations  (pilots,  combat 
system  operators,  air  battle  managers)  (Carretta,  2008;  Carretta  &  Ree,  2003;  Olea  &  Ree,  1994). 
This  body  of  knowledge  provides  a  firm  foundation  for  delineating  selection  methods  and  tools 
for  the  new  RPA  career  fields. 
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When  we  began  this  project  in  September  2009,  a  great  deal  of  work  had  already  been 
conducted  to  identify  RPA  job  requirements  and  more  was  underway  (e.g.,  Tvaryanas,  2006; 

UAS  Operator  Way  Ahead,  2008).  Front-end  analyses  were  available  for  Predator  pilots  (Kalita 
&  Duma,  2008a;  Nagy,  Kalita,  &  Eaton,  2006a)  and  Predator  SOs  (Kalita  &  Duma,  2008b;  Nagy, 
Eaton,  &  Muse,  2006b).  However,  these  analyses  were  based  on  small  samples  and  did  not 
identify  detailed  SAOC  requirements.  There  were  also  several  ongoing  initiatives  focusing  on 
RPA  selection,  including  efforts  conducted  by  the  711  HPW/HP  and  USAF  School  of  Aerospace 
Medicine  (USAFSAM/FEC),  both  at  Brooks  City  Base,  TX,  and  efforts  at  71 1  HPW/RHAS  at 
Mesa,  AZ.  For  example,  711  HPW/RHA  had  recently  completed  Mission  Essential  Competency 
(MEC)  analyses  for  the  Predator  and  Reaper  RPAs.  The  MECs  for  these  two  RPAs  detail 
competencies  exhibited  by  successful  personnel  in  an  operational  environment,  including,  an 
Initial  Competency  Set  (ICS)  that  is  the  foundation  for  learning  to  operate  Predator  and  Reaper 
RPAs.  (Additional  training  and  development  would  be  required  to  master  all  operational  MECs.) 

USAFSAM/FEC  also  had  conducted  several  relevant  activities.  First,  it  developed  a 
profde  for  successful  RPA  pilots  and  SOs  based  on  interviews  and  focus  groups  conducted  with 
RPA  personnel  (e.g.,  Wing  CCs,  Squadron  CCs  and  DOs,  Pilots,  and  SOs)  (personal 
communication,  Dr.  Wayne  Chappelle,  October  30,  2009).  This  profile  initially  comprised  125 
attributes  grouped  into  12  broader  domains.  The  initial  set  of  attributes  was  then  distilled  into  16 
critical  attributes  covering  4  domains  (both  cognitive  and  noncognitive).  USAFSAM/FEC  had 
also  met  with  United  Kingdom  (UK)  Royal  Air  Force  (RAF)  scientists  concerning  the  RAF’s  job 
analysis  of  their  RPA  pilot  and  SO  positions.  The  UK  RAF  was  in  the  process  of  developing  and 
validating  a  RPA  selection  battery  (Bailey,  2008). 

The  Air  Force  Personnel  Center,  Research  and  Analysis  Branch  (hereafter  AFPC), 
required  technical  assistance  to  review  and  qualitatively  analyze  information  available  from 
these  earlier  and  ongoing  efforts,  and  to  map  this  information  against  existing  taxonomies  of 
SAOCs  to  determine  if  there  are  any  gaps  in  coverage.  If  any  gaps  were  identified,  AFPC 
requested  technical  assistance  in  identifying  potential  measures  of  the  missing  SAOCs  or, 
possibly,  efforts  to  develop  such  measures.  The  overall  goal  of  these  analyses  was  to  help  AFPC 
identify  a  reasonable  number  of ’’best  bet”  measures  for  assessing  critical  entry-level  SAOCs  for 
RPA  pilot  and  SO  positions  in  the  Active  Duty  USAF. 

The  UK  RAF  Predator  selection  battery  was  of  particular  appeal  in  this  effort  because  it 
was  developed  specifically  for  the  population  of  interest.  Unfortunately,  it  became  apparent 
within  a  few  months  of  contract  award  that  we  would  not  be  able  to  do  a  focused  study  of  the  test 
battery  because  (a)  some  portions  were  still  under  development  and  (b)  the  UK  RAF  was  not  yet 
ready  to  engage  in  a  collaborative  study  with  AFPC.  Consequently,  the  AFPC  project  team 
directed  HumRRO  to  explore  other  potential  predictor  measures,  including  those  already  being 
used  for  selection  into  RPA  pilot/SO  training  or  similar  jobs  and  those  targeting  SAOCs  that  job 
analytic  work  suggested  were  important  for  the  RPA  pilot  and/or  SO  job. 

Approach 

The  AFPC  project  team  provided  access  to  several  technical  reports  and  briefing  slides 
that  touched  on  RPA  pilot  and  SO  job  duties  in  the  USAF  and  the  SAOCs  required  to  perform 
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those  duties.  HumRRO  supplemented  these  source  documents  through  key  word  searches  in 
information  repositories  such  as  the  Defense  Technical  Information  Center  (DTIC)  and  through 
our  own  contacts  in  the  U.S.  Army  and  the  USAF.  Primary  sources  included: 

•  MECs  for  Predator,  Reaper,  and  Global  Hawk  RPA  platforms/missions  provided  by  the 
Air  Force  Research  Laboratory,  Warfighter  Readiness  Research  Division  (711 
HPW/RHAS)1 

•  In-progress  information  about  job  analytic  research  being  conducted  by  USAFSAM 
(personal  communication,  Dr.  Wayne  Chappelle,  October  30,  2009;  Chappelle, 
McDonald,  &  King,  20 102;  Chapelle,  McDonald,  &  McMillan,  2011) 

•  Front  End  analysis  of  Predator  pilot  and  SO  job  requirements  (Kalita  &  Duma,  2008a; 
2008b;  Nagy,  Eaton,  &  Muse,  2006b;  Nagy,  Kalita,  &  Eaton,  2006a) 

•  AF  Briefing  Slides  (4  August  2008).  UAS  Operator  Way  Ahead  provided  by  the  AFPC 
project  team. 

•  Tvaryanas  (6  August  2006).  Unmanned  aircraft  systems  (UAS)  skill  sets.  Briefing  slides 
provided  by  the  AFPC  project  team. 


Description  of  RPA  Work  Context  and  Job  Activities  Provided  in  Source  Documents  and  by 

USAF  Subject  Matter  Experts  (SMEs) 

Figure  1  shows  two  images  of  ground  control  stations  (GCS)  used  by  RPA  pilots  and  SOs. 
The  configuration  of  the  GCS  can  and  does  vary  from  one  unit  to  another,  and  these  images  are 
likely  already  at  least  somewhat  outdated.  Still,  the  images  illustrate  what  subject  matter  experts 
(SMEs)  and  the  sources  above  conveyed:  all  GCS  involve  multiple  sources  of  visual  information 
displayed  on  several  two-dimensional  high-resolution  monitors.  Visual  information  comes  in  the 
form  of  terrain  maps  and  satellite  images,  text,  numerical  information,  and  dashboard  analogs  of  the 
gauges  or  dials  that  might  appear  in  the  cockpit  of  a  manned  aircraft.  All  GCS  configurations  also 
include  multiple  modes  of  communication,  including  information  presented  through  a  headset,  e-mail 
and  chat  capabilities,  and  a  landline  telephone.  Within  these  modes  of  communication,  there  can  be 
multiple  streams  of  information,  for  example,  several  different  chat  rooms,  oral  communication  with 
several  different  parties  over  the  headset,  and  more  than  one  line  on  the  landline  telephone.  SMEs 
infonned  us  that  multi-tasking  is  occurring  most  of  the  time,  in  the  sense  that  there  is  more  than 
one  channel  of  information  to  be  attended  to  and  processed.  There  is  variability  in  the  number  of 
sources  and  amount  of  information  coming  in  at  once,  the  rate  at  which  things  are  happening,  and 
the  level  of  consequences  associated  with  multi-tasking  perfonnance.  At  the  lowest  intensity 
levels,  RPA  pilots  and  SOs  can  easily  shift  attention  from  one  display  to  another  at  a  relatively 
slow  pace  and  with  little  negative  impact  if  they  fail  to  notice  something.  At  the  highest  intensity 
levels,  the  infonnation  overload  becomes  very  difficult  to  handle,  but  the  stakes  are  very  high 


1  The  MECs  are  not  available  for  general  public  distribution  so  are  not  listed  in  this  report. 

"  In  2009,  Dr.  Wayne  Chappelle  provided  information  about  SAOC  requirements  from  ongoing  research.  We  did  not 
have  access  to  the  2010  or  2011  reports  cited  here  when  we  were  identifying  SAOC  requirements 
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(e.g.,  failure  to  protect  friendly  forces  or  loss  of  the  aircraft).  During  times  of  information  overload,  it  is 
critical  for  RPA  pilots  and  SOs  to  realize  that  they  are  becoming  task  saturated  and  to  tell  others  so  they 
can  help.  The  images  in  Figure  1  helped  us  understand  the  perceptual  and  cognitive  load  placed  on  RPA 
pilots  and  SOs. 


Figure  1.  Images  of  the  ground  control  station  (GCS). 
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Following  is  a  summary  of  how  a  “typical”  shift  might  play  out  for  an  RPA  crew,  as 
described  by  several  experienced  RPA  pilots  and  SOs.  At  the  beginning  of  the  shift,  the  RPA 
pilot  and  SO  nest  in  to  their  workstation,  often  taking  over  from  a  crew  that  has  been  flying  the 
RPA  for  the  preceding  work  shift.  After  nesting  in,  there  is  often  a  period  of  minutes  or  hours  of 
relatively  little  activity  as  the  RPA  flies  to  its  final  destination,  or  as  it  circles  an  area  where  it  is 
providing  surveillance  and  not  much  is  happening.  During  this  time,  the  crew  must  monitor  the 
(a)  status  of  the  RPA,  (b)  chat  room  messages,  (c)  radio  transmissions  that  might  be  relevant  for 
them,  (d)  changes  in  the  airspace,  and  (e)  other  aircraft  operating  in  the  same  or  nearby  airspace. 
During  these  periods,  maintaining  vigilance  is  a  bigger  challenge  than  worrying  about 
information  overload.  This  can  be  especially  hard  when  the  crew  is  working  a  night  shift  or  is 
fatigued.  The  crew  can  use  the  slow  time  to  do  lower  priority  tasks  such  as  completing  flight  logs 
or  other  administrative  paperwork.  Often  the  RPA  must  pass  through  or  around  airports  and/or 
Restricted  Operating  Zones  (ROZ)  on  its  way  to  the  target  destination,  so  the  RPA  crew  must 
coordinate  the  airspace  as  they  go.  As  soon  as  the  RPA  arrives  in  the  target  airspace,  the  crew 
usually  encounters  other  air  traffic  in  that  space  so  airspace  coordination  must  continue. 

A  fairly  common  occurrence  that  involves  a  moderate  degree  of  intensity  is  flying  along 
“fat,  dumb,  and  happy”  in  a  particular  airspace  and  then  being  notified  that  the  RPA  can  no 
longer  be  in  that  airspace.  At  this  point,  there  likely  will  be  other  aircraft  stacked  in  the  area-as 
close  as  every  500  feet  vertically.  When  this  happens,  the  pilot  must  contact  Air  Traffic  Control 
(ATC)  to  request  a  move  to  a  new  airspace.  This  can  take  20  minutes  of  searching  and 
communicating.  Another  fairly  common  occurrence  is  that  the  mission  rapidly  changes  from 
quiet  and  boring  to  very  intense.  Flere,  the  most  common  example  is  when  friendly  forces  come 
under  fire  (or  encounter  a  roadside  bomb).  Suddenly,  the  RPA  crew  experiences  a  “fire  hose”  of 
information.  There  is  typically  a  Joint  Terminal  Attack  Controller  (JTAC)  on  the  ground  calling 
for  assistance  who  may  be  quite  emotional.  Several  other  people  in  the  area  typically  also  start 
communicating  on  the  radio  and/or  by  chat,  for  example,  other  nearby  ground  forces,  other 
aircraft  in  the  area,  and  military  commanders  who  are  in  the  chain  of  command  of  the  unit  under 
fire.  At  this  intense  time,  the  SO  often  must  concentrate  on  keeping  the  camera  on  an  identified 
target,  while  also  maintaining  enough  situational  awareness  to  help  the  pilot  keep  the  RPA  in  the 
right  position  and  be  alert  for  potential  problems  (e.g.,  coming  too  close  to  other  aircraft  in  the 
area).  The  target  could  be  fixed  or  moving.  Ideally,  the  SO  will  be  thinking  ahead  to  how  the 
aircraft  will  need  to  be  positioned  in  order  to  maintain  a  good  camera  angle  on  the  target,  and 
conveying  this  information  to  the  pilot.  The  camera  cannot  take  a  picture  of  something  that  is 
directly  below  the  aircraft.  Therefore,  if  the  pilot  does  not  keep  the  aircraft  in  a  good  position,  the 
SO  may  lose  visual  contact.  The  SO  cannot  prevent  this  from  happening,  except  to  help  the  pilot 
avoid  flying  directly  over  the  target.  While  the  SO  is  focusing  on  keeping  the  camera  on  the 
target,  the  Pilot  attempts  to  keep  the  aircraft  in  the  best  position  to  maintain  visual  contact  with 
the  target  and  tries  to  “stay  ahead  of  the  aircraft”  so  they  can  anticipate  where  the  aircraft 
should  best  be  positioned  in  the  future.  At  the  same  time,  the  pilot  needs  to  maintain  an 
awareness  of  other  aircraft  in  the  area,  and  other  ground  forces  or  combatants  that  may  start 
firing  their  weapons.  The  pilot  needs  to  get  permission  from  ATC  to  open  up  a  ROZ,  and  these 
requirements  may  change  during  the  mission,  especially  when  tracking  a  moving  object.  The 
pilot  does  this  by  calling  ATC.  The  pilot  also  is  monitoring  communications  from  several 
sources,  including  the  JTAC,  the  pilot’s  own  RPA  Mission  Commander  (MC),  other  units  in 
the  area,  and  so  on.  Both  the  pilot  and  SO  can  delegate  some  tasks  to  the  MC  (e.g.,  monitoring 
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chat  from  some  of  the  sources  or  setting  up  and  gaining  access  to  a  ROZ).  Pilots  vary  in  how 
much  they  are  willing  to  delegate  responsibility,  depending  on  their  assessment  of  how  much 
they  trust  the  SO  or  MC  to  handle  the  delegated  tasks.  To  make  this  kind  of  situation  even  more 
intense,  there  could  be  a  problem  with  the  satellite  link  or  with  the  RPA  itself  When  this 
happens,  maintenance  crews  will  be  in  the  GCS  working  on  the  equipment.  Finally,  the  pilot  may 
need  to  notice  and  respond  to  an  aircraft  status  indicator  while  trying  to  do  all  of  the  above. 

Other  Sources  of  Information  about  SAOC  Requirements 

In  addition  to  compiling  the  preceding  word  picture  of  the  RPA  work  context  and  work 
activities,  FIumRRO  expanded  the  search  to  include  as  much  information  as  we  could  find  about 
duty  and  SAOC  requirements  for  similar  jobs,  including  the  following: 

•  RPA  pilots  other  than  those  in  the  USAF,  including  the  UK  RAF,  U.S.  Army, 
and  NASA.  Some  of  these  documents  included  formal  studies  of  job 
requirements;  many  were  popular  press  articles  describing  job  duties  and  SAOC 
requirements  with  varying  degrees  of  detail  and  rigor  (Bailey,  2008;  Barnes, 
Knapp,  Tillman,  Walters  &  Velicki,  2000;  Biggerstaff,  Blower,  Portman,  & 
Chapman,  1998;  Bruskiewicz,  Houston,  Hezlett,  &  Ferstl,  2007;  Kay,  Dolgin, 
Wasel,  Langelier  &  Hoffman,  1999;  Montijo,  Kaiser,  Spiker,  &  Nullmeyer, 

2008;  Pestana,  2007). 

•  Pilots  of  military  manned  aircraft,  including  both  fixed  and  rotary  wing  (Agee, 
Shore,  Alley,  Barto,  &  Halper  2009;  Bruskiewicz,  Katz,  Houston,  Paullin,  O’Shea,  & 
Damos  2007;  Driskill,  Koonce,  Nance,  &  Weissmuller  2001;  Kubisiak  &  Katz, 

2006;  Mangos,  Arnold,  Mead,  Merket,  Littrell,  Credo,  Young,  Tolentino,  &  Kessler, 
2005;  Paullin,  Katz,  Bruskiewicz,  Houston,  &  Damos,  2006). 

Given  that  our  charge  was  to  identify  entry-level  selection  requirements,  we  focused  most 
of  our  efforts  on  identifying  SAOC  requirements,  and  less  on  delineating  specific  job  tasks.  In 
fact,  the  MECs  capture  all  critical  duties  at  a  sufficient  level  of  detail  to  infer  SAOC 
requirements,  as  judged  by  RPA  SMEs.  The  UK  RAF  Predator  study  (Bailey,  2008)  also 
identified  a  set  of  duty  requirements  very  similar  to  the  MECs.  Furthermore,  both  the  MEC  and 
UK  RAF  studies  concluded  that  there  is  little  difference  in  duty  requirements  across  the  MQ- 1 
(Predator)  and  MQ-9  (Reaper)  aircraft  that  are  most  typically  flown  by  USAF  personnel,  and 
relatively  little  difference  in  duty  requirements  between  the  RPA  pilot  and  the  SO. 

Each  group  of  researchers  used  its  own  labels  and  its  own  organizing  structure  for  the 
SAOCs,  so  one  of  our  challenges  was  finding  a  way  to  integrate  all  of  the  existing  information 
into  a  meaningful  organizational  framework.  We  wanted  a  framework  that  would  lead  to 
differentiable  and  measurable  individual  differences  constructs,  and  one  that  is  widely  used,  so 
we  turned  to  the  U.S.  Department  of  Labor’s  Occupational  Information  Network  (0*NET). 
Mangos  et  al.  (2005)  followed  a  similar  procedure  when  analyzing  job  requirements  for  Navy 
aviators. 

0*NET  is  a  comprehensive  system  developed  for  the  U.S.  Department  of  Labor  that 
provides  information  for  more  than  one  thousand  occupations  within  the  U.S.  economy.  This 
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information  is  maintained  in  a  comprehensive  database  which  was  developed  to  replace  the 
Dictionary  of  Occupational  Titles  (DOT)  (U.S.  Department  of  Labor,  1991).  This  taxonomy  is 
organized  hierarchically  and  is  based  on  decades  of  taxonomic  research  in  the  arenas  of  abilities, 
skills,  personality  traits  (which  are  called  Work  Styles  in  the  0*NET  framework),  and  work 
context  factors.  The  0*NET  Content  Model  was  developed  based  on  a  thorough  review  of  an 
extensive  body  of  literature  from  the  job  analysis  arena  within  the  field  of  industrial- 
organizational  psychology  (McCloy,  Campbell,  Oswald,  Lewis,  &  Rivkin,  1999;  Peterson, 
1999).  The  0*NET  Content  Model  contains  four  types  of  descriptors:  job-oriented,  worker- 
oriented,  cross-occupational,  and  occupation-specific.  In  turn,  these  descriptors  are  organized 
into  the  six  domains  shown  in  Figure  2.  For  purposes  of  this  project,  we  focused  on  Worker 
Characteristics  that  would  be  present  at  entry  to  an  RPA  career  field,  in  keeping  with  our  charge 
to  focus  on  entry-level  requirements.  We  also  examined  one  aspect  of  the  Occupational 
Requirements  domain-Work  Context-  because  the  RPA  work  environment  involves  several 
salient  and  fairly  unique  context  factors,  including  flying  battle  missions  from  a  remote  location 
and  experiencing  much  of  the  emotional  stress  that  accompanies  manned  flight  without  the 
physical  and  physiological  characteristics  of  manned  flight. 
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Figure  2.  0*NET  content  model. 
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Compiling  an  SAOC List 


We  initially  compiled  a  comprehensive  list  of  10  Skills,  25  Abilities,  and  12  Work  Styles. 
The  list  can  be  found  in  Appendix  A.  The  HumRRO  team  applied  its  extensive  knowledge  of 
individual  differences  taxonomies,  and  the  0*NET  taxonomy  in  particular,  to  map  SAOCs  listed 
in  various  reports  onto  the  0*NET  Content  Model.  Each  of  the  selected  SAOCs  was  mentioned 
as  important  for  performing  the  RPA  pilot  and/or  SO  job,  or  a  similar  job  (i.e.,  manned  aircraft 
pilot)  in  one  or  more  of  the  sources  cited  above.  The  one  concept  that  did  not  appear  in  the 
0*NET  content  model  but  was  included  in  our  initial  list  was  “situational  awareness.”  This  skill 
is  commonly  cited  by  aviators  as  a  critical  skill,  so  it  was  included  in  the  list  at  this  point. 

Next,  we  asked  several  people  knowledgeable  about  the  USAF  RPA  pilot  and  SO  jobs  to 
review  the  list.  The  SMEs  are  shown  in  Table  1.  As  a  group,  they  had  experience  in  screening  for 
manned  and  RPA  pilot  training,  developing  and  delivering  training  for  RPA  pilots  and  SOs, 
establishing  the  RPA  SO  career  field,  and  developing  RPA  training  simulators.  Several  SMEs 
also  had  experience  flying  RPAs.  During  each  meeting,  we  asked  the  SMEs  to  comment  on  the 
SAOC  list  in  general  and  to  identify  the  SAOCs  that  differentiated  unmanned  from  manned 
flight  or  that  differentiated  RPA  pilots  from  SOs.  After  the  SMEs  had  reviewed  and  discussed 
the  entire  list,  we  also  asked  them  if  any  critical  SAOCs  were  missing.  In  those  cases  where 
SMEs  identified  missing  SAOCs,  we  were  typically  able  to  address  their  suggestion  by  slightly 
modifying  the  definition  for  an  existing  SAOC  rather  than  adding  entirely  new  ones  to  the  list. 

Table  2  contains  the  initial  set  of  29  SAOCs  judged  to  be  important  for  perfonning  the 
RPA  pilot  and  SO  jobs.  It  is  worth  noting  that  none  of  the  SMEs  felt  there  were  any  significant 
differences  between  the  pilot  and  SO  jobs  in  terms  of  fundamental  SAOC  requirements.  Next, 
we  narrowed  the  set  to  2 1  SAOCs  based  on  SME  judgments  regarding  which  are  the  most 
critical,  and  these  are  shown  in  Table  3.  We  also  took  into  account  the  trainability  of  the  SAOCs 
because  SAOCs  that  can  be  easily  trained  represent  less  critical  targets  for  selection  purposes. 
These  were  consensus  judgments  based  on  independent,  qualitative  input  from  several  iterations 
of  review  by  USAF  SMEs  and  our  own  project  staff. 
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Table  1.  Subject  Matter  Experts  (SMEs)  Who  Reviewed  RPA  SAOC  Lists 


Name 

Affiliation/Unit 

Type  of  Expertise 

Ray  King,  LtCol 

USAE  AFMC  USAESAM/FEC 

RPA  Pilot  screening  &  selection 

Wayne  Chappelle,  Ph.D.,  Civ 

USAF  AFMC  USAFSAM/FEC 

RPA  Pilot  screening  &  selection 

Victor  Allen,  CMSgt 

USAF  AF/A30-AT 

RPA  SO  career  field  management 

Baylen  Johnson,  SMSgt 

USAF  AF/A30-AT 

RPA  SO  career  field  management 

Michael  Elson,  SMSgt 

USAF  AF/A30-AT 

RPA  SO  career  field  management 

Cami  Anderson,  MSgt 

USAF  AF/A30-AT 

RPA  SO  career  field  management 

John  Gillis,  Civ 

USAF  AETC/A3F 

RPA  Pilot  training 

Jeffrey  Wiseman,  Civ 

USAF  AETC/A3FR 

Chief,  AETC  RPA  Training 

Branch 

Russell  Gamer,  LtCol 

USAF  AETC  12  OG/DET  1/DO 

RPA  operator  training 

Mark  Hand,  Maj 

USAF  AETC  558  FTS/RFC 

RPA  operator  training 

Thomas  McCurley,  LtCol 

USAF  ACC  16  TRS/DO 

RPA  operator  training 

Robert  Englehart,  Civ 

USAF  AETC/A3FR 

Deputy  Branch  Chief, 

HQ  AETC  RPA  Training 

Pablo  Sanchez,  CTR 

USAF  AFMC  711  HPW/RHA 

Pilot  Training 

Daniel  Walker,  Col 

USAF  AFMC  711  HPW/RHA 

RPA  Pilot  Training  and 
operations 

Thomas  Carretta,  Ph.D. 

USAF  AFMC  711  HPW/RHCI 

Expert  on  USAF  Selection 

Diane  Damos,  Ph.D. 

Damos  Aviation  Services 

Aviation  Psychologist  who 
consults  on  pilot  selection  and 
training  for  military  and  civilian 
clients 

William  Howse,  Ph.D. 

Damos  Aviation  Services 

Formerly  Chief  of  the  Rotary 

Wing  Aircraft  Research  Unit, 

U.S.  Army  Research  Institute  for 
the  Behavioral  and  Social 

Sciences;  currently  with  Damos 
Aviation  Services 
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Table  2.  List  of  RPA  Pilot  and  SO  SAOC  requirements  after  SME  review 


Type 

Construct 

0*NET  Definition 

Work  Style 

Initiative 

Willingness  to  take  on  responsibilities  and  challenges. 

Work  Style 

Leadership 

(Assertiveness) 

Willingness  to  speak  up  and  offer  suggestions,  recommendations,  or 
opinions,  even  if  others  may  not  respond  favorably. 

Work  Style 

Cooperation 

Being  pleasant  with  others  on  the  job  and  displaying  a  good-natured, 
cooperative  attitude. 

Work  Style 

Self  Control 

Maintaining  composure,  keeping  emotions  in  check,  even  in  very  difficult 
situations;  quickly  refocusing  attention  on  task  after  making  an  error  or 
witnessing  an  emotionally  disturbing  event. 

Work  Style 

Stress  Tolerance 

Accepting  criticism  and  dealing  calmly  and  effectively  with  high  stress 
situations. 

Work  Style 

Adaptability/Flexibility 

Being  open  to  change  (positive  or  negative)  and  to  considerable  variety  in 
the  workplace. 

Ability 

Oral  Comprehension 

The  ability  to  understand  information  presented  orally  in  a  variety  of 
conditions,  including  situations  in  which  multiple  parties  are 
communicating  or  under  conditions  of  stress. 

Ability 

Written  Comprehension 

The  ability  to  read  and  understand  information  and  ideas  presented  in 
writing. 

Ability 

Oral  Expression 

The  ability  to  communicate  information  and  ideas  in  speaking  so  others 
will  understand;  the  ability  to  use  standard  codes  to  convey  information 
orally. 

Ability 

Deductive  Reasoning 

The  ability  to  apply  general  rules  to  specific  problems  to  produce  answers 
that  make  sense. 

Ability 

Inductive  Reasoning 

The  ability  to  combine  pieces  of  information  to  form  general  rules  or 
conclusions  (includes  finding  a  relationship  among  seemingly  unrelated 
events). 

Ability 

Information  Ordering 

The  ability  to  arrange  things  or  actions  in  a  certain  order  or  pattern 
according  to  a  specific  rule  or  set  of  rules  (e.g.,  patterns  of  numbers, 
letters,  words,  pictures,  mathematical  operations). 

Ability 

Number  Facility 

The  ability  to  recognize  and  process  numbers  quickly  and  accurately, 
including  performing  basic  mathematical  operations  without  the  use  of 
external  aids. 

Ability 

Working  Memory 

The  ability  to  temporarily  hold  information  in  memory  while  processing 
other  information. 
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Table  2.  (Continued) 


Type 

Construct 

0*NET  Definition 

Ability 

Speed  of  Closure 

The  ability  to  quickly  make  sense  of,  combine,  and  organize  info 
in  a  meaningful  way. 

Ability 

Flexibility  of  Closure 

The  ability  to  identify  or  detect  a  known  pattern  (a  figure,  object,  word,  or 
sound)  that  is  hidden  in  other  distracting  material. 

Ability 

Perceptual  Speed 

The  ability  to  quickly  and  accurately  compare  similarities  and  differences 
among  sets  of  letters,  numbers,  objects,  pictures,  or  patterns.  The  things  to 
be  compared  may  be  presented  at  the  same  time  or  one  after  the  other. 

This  ability  also  includes  comparing  a  presented  object  with  a  remembered 
object. 

Ability 

Spatial  Orientation 

The  ability  to  know  your  location  in  relation  to  the  environment  or  to 
know  where  other  objects  are  in  relation  to  you. 

Ability 

Visualization 

The  ability  to  imagine  how  something  will  look  after  it  is  moved  around  or 
when  its  parts  are  moved  or  rearranged. 

Ability 

Selective  Attention 

The  ability  to  concentrate  on  a  task  over  a  period  of  time  without  being 
distracted. 

Ability 

Time  Sharing 

The  ability  to  shift  back  and  forth  between  two  or  more  activities  or 
sources  of  information  (such  as  speech,  sounds,  touch,  or  other  sources). 

Ability 

Situational  Awareness 

The  ability  to  track  changing  information  and  events  in  a  dynamic 
environment  and  assess  their  impact  on  the  mission.  [NOTE:  This  ability 
is  not  part  of  the  0*NET  taxonomy] 

Ability 

Control  Precision 

The  ability  to  quickly  and  repeatedly  adjust  the  controls  of  a  machine  or  a 
vehicle  to  exact  positions. 

Ability 

Rate  Control 

The  ability  to  time  your  movements  or  the  movement  of  a  piece  of 
equipment  in  anticipation  of  changes  in  the  speed  and/or  direction  of  a 
moving  object  or  scene. 

Skill 

Critical  Thinking 

Skilled  at  using  logic  and  reasoning  to  identify  the  strengths  and 
weaknesses  of  alternative  solutions,  conclusions  or  approaches  to 
problems. 

Skill 

Complex  Problem 
Solving 

Skilled  at  identifying  complex  problems  and  reviewing  related 
information  to  develop  and  evaluate  options  and  implement 
solutions. 

Skill 

Judgment  and  Decision 
Making 

Skilled  at  considering  the  relative  costs  and  benefits  of  potential  actions 
to  choose  the  most  appropriate  one. 

Skill 

Coordination 
(T  eamwork) 

Skilled  at  adjusting  actions  in  relation  to  others'  actions. 
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Situational  Awareness.  At  this  point,  we  had  not  reached  a  final  conclusion  on  whether 
to  include  situational  awareness  in  the  list  of  critical  SAOC.  Ultimately  it  was  dropped  as  a  target 
for  measurement  in  the  entry-level  selection  process.  While  situational  awareness  can  be 
measured  reliably  (Carretta,  Perry,  &  Ree,  1996;  Endsley  &  Bolstad,  1994;  Matthews  &  Beal, 
2002),  it  appears  to  be  a  function  of  underlying  cognitive  processing  abilities,  several  of  which 
are  already  represented  in  the  critical  SAOC  list,  and  knowledge  or  experience  in  a  particular 
context  or  domain.  For  example,  Carretta  et  al.  (1996)  found  that  flying  experience  was  the  best 
predictor  of  situational  awareness  among  F-15  pilots.  In  this  study,  situational  awareness  was 
measured  through  observer  (peer  and  supervisor)  ratings.  The  same  study  found  that  scores  on 
measures  of  general  cognitive  abilities  including  working  memory,  spatial  reasoning,  and 
divided  attention  were  predictive  of  observer  ratings  of  pilot  situational  awareness.  Endsley 
(1995)  developed  a  theory  of  situational  awareness  and  defines  it  as  a  state  of  knowledge.  She 
differentiates  it  from  the  cognitive  processes  used  to  achieve  it.  She  further  theorizes  that  factors 
such  as  attention,  working  memory,  workload,  and  stress  can  influence  situational  awareness, 
and  that  situational  awareness  is  more  than  the  sum  of  these  concepts.  Bedney  and  Meister 
(1999)  argue  that  Endsley’s  theory  is  logically  flawed,  but  agree  with  her  that  situational 
awareness  is  not  a  single  individual  differences  construct.  They  describe  it  as  “part  of  a  cognitive 
activity  that  is  intensely  dynamic.” 

Situational  awareness  has  been  measured  using  rating  scales  to  collect  self  or  observer 
ratings  (see  Bell  &  Lyon,  2000  for  a  review;  also  Carretta  et  al.,  1996;  Matthews  &  Beal,  2002). 
These  measures  are  typically  administered  after  target  subjects  perform  some  complex  activity, 
for  example,  after  a  training  flight  or  after  a  field  training  exercise.  It  would  be  very  difficult  to 
collect  observer  ratings  in  an  entry-level  selection  process  and  self-ratings  could  be  easily  faked. 
Situational  awareness  also  has  been  measured  as  part  of  multi-tasking  tests  and  simulations 
(Endsley  &  Bolstad,  1994;  O’Brian  &  O’Hare,  2007;  O’Hare,  1997;  Ramos,  Heil,  &  Manning, 
2001a,  2001b).  More  information  is  provided  about  these  measures  in  Appendix  B.  Some  of 
them  treat  situational  awareness  as  the  composite  of  performance  on  several  tasks  that  occur 
concurrently  (e.g.,  Wombat©;  O’Brian  &  O’Hare,  2007;  O’Hare,  1997).  Other  measures  attempt 
to  evaluate  situational  awareness  by  stopping  a  simulation  in-progress  and  posing  questions 
about  the  state  of  affairs  at  that  point  in  time  (e.g.,  Air  Traffic  Scenarios  or  Letter  Factory  Test  in 
the  FAA  Air  Traffic  Selection  and  Training  Battery  [ATSAT];  Ramos,  Heil,  &  Manning,  2001a, 
2001b).  The  questions  might  ask  the  test-taker  to  identify  which  objects  were  present  or  where 
they  were  located  in  the  simulated  space,  the  number  of  objects  of  different  types  that  were 
present,  the  status  of  one  or  more  indicators  (such  as  a  fuel  gauge),  and  so  on.  Critics  of  the 
latter  approach  argue  that  situational  awareness  is  more  than  the  ability  to  answer  questions 
about  information  being  held  in  working  memory. 

Ultimately,  we  discarded  the  idea  of  recommending  a  measure  of  situational  awareness  in 
the  entry-level  selection  process.  However,  we  do  believe  that  critical  aspects  of  it  can  be 
measured  using  a  multi-tasking  measure  that  requires  application  of  working  memory,  time 
sharing,  and  task  prioritization.  This  belief  is  supported,  at  least  to  some  degree,  by  research  on 
situational  awareness  among  pilots  (Carretta  et  al.,  1996;  Endsley  &  Bolstad,  1994).  For 
purposes  of  an  entry-level  selection  battery,  we  detennined  that  it  would  be  most  appropriate  to 
focus  on  basic  abilities  that  underlie  situational  awareness,  though  they  likely  do  not  fully 
capture  this  important  ability. 
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Table  3.  SAOCs  Judged  to  Be  Most  Critical  to  RPA  Pilot  and  SO  Occupations 


Type 

Attribute 

Definition 

Non-Cog 

Initiative 

To  initiate  tasks  or  duties,  even  difficult  ones,  without  excessive 
procrastination;  to  work  independently  and  accomplish  tasks 
without  constant  supervision;  to  take  personal  responsibility  for 
completing  work  assignments. 

Non-Cog 

Assertiveness 

To  speak  up  and  to  offer  suggestions,  recommendations,  or 
opinions  when  appropriate,  even  if  others  may  not  respond 
favorably. 

Non-Cog 

Decisiveness 

To  make  good  decisions  rapidly,  based  on  available  information. 

Non-Cog 

Self  Control 

To  maintain  composure  and  keep  emotions  in  check,  even  in  very 
difficult  situations;  to  quickly  refocus  attention  on  the  primary  task 
after  making  an  error  or  witnessing  an  emotionally  disturbing 
event. 

Non-Cog 

Stress  Tolerance 

To  remain  calm,  analyze  the  situation,  act  appropriately,  and  make 
quick,  accurate  decisions  in  high  workload,  time  pressure,  or 
other  stressful  situations. 

Non-Cog 

Adaptability 

To  adjust  easily  to  change  in  situations  or  unexpected  events;  to 
flexibly  change  one's  actions  in  response  to  changing  task 
priorities. 

Ability 

Oral  Comprehension 

The  ability  to  understand  information  presented  orally  in  a 
variety  of  conditions,  including  situations  in  which  multiple 
parties  are  communicating  or  under  conditions  of  stress. 

Ability 

Oral  Expression 

The  ability  to  communicate  information  and  ideas  in  speaking  so 
others  will  understand;  the  ability  to  use  standard  codes  to  convey 
information  orally. 

Ability 

Number  Facility 

The  ability  to  recognize  and  process  numbers  quickly  and 
accurately,  including  performing  basic  mathematical 
operations  without  using  external  aids  (e.g.,  calculator). 

Ability 

Working  Memory 

The  ability  to  temporarily  hold  information  in  memory  while 
processing  other  information. 

Ability 

Task  Prioritization 

The  ability  to  perform  multiple  tasks  in  order  of  their  importance; 
to  direct  attention  to  tasks  when  they  change  priorities  (e.g., 
emergencies). 

Ability 

Selective  Attention 

The  ability  to  maintain  high  levels  of  concentration  on  a  task  in 
distracting  or  repetitive  conditions;  to  maintain  focus  despite 

interruptions. 

13 


Table  3.  (Continued) 


Type 

Attribute 

Definition 

Ability 

Time  Sharing 

The  ability  to  flexibly  switch  attention  across  different  tasks; 
to  attend  to  multiple,  potentially  conflicting  sources  of 
information. 

Ability 

Perceptual  Speed 

The  ability  to  quickly  and  accurately  compare  similarities  and 
differences  among  sets  of  letters,  numbers,  objects,  pictures,  or 
patterns.  The  things  to  be  compared  may  be  presented  at  the  same 
time  or  one  after  the  other.  This  ability  also  includes  comparing  a 
presented  object  with  a  remembered  object. 

Ability 

Spatial  Orientation 

The  ability  to  know  one's  location  in  relation  to  the  environment 
or  to  know  where  other  objects  are  in  relation  to  oneself. 

Ability 

Visualization 

The  ability  to  imagine  how  something  will  look  after  it  is  moved 
around  or  when  its  parts  are  moved  or  rearranged. 

Ability 

Pattern  Recognition 

The  ability  to  identify  or  detect  a  known  pattern  (for  example,  a 
numerical  code);  to  combine  and  organize  different  pieces  of 
information  into  a  meaningful  pattern  quickly. 

Ability 

Control  Precision 

The  ability  to  control  the  motion  of  a  machine,  vehicle,  or 
piece  of  equipment  (e.g.,  joystick  or  yoke)  quickly  and 
accurately;  to  make  fine,  precise  movements  or  adjustments. 

Skill 

Critical  Thinking 

Skilled  at  using  logic  and  reasoning  to  identify  the  strengths  and 
weaknesses  of  alternative  solutions,  conclusions  or  approaches  to 
problems. 

Skill 

Judgment  and 
Decision  Making 

Skilled  at  considering  the  relative  costs  and  benefits  of 
potential  actions  before  choosing  the  most  appropriate  action. 

Skill 

Teamwork  Skills 

Skilled  at  coordinating  with  others  in  a  team  setting  to  accomplish 
group  goals;  sharong  information  to  ensure  shared  understanding 
of  the  mission  and  situation;  assisting  team  members  as  needed  to 
ensure  mission  success. 

Describing  Key  Work  Context  Factors 

Adapting  to  the  RPA  work  context  is  a  critical  aspect  of  the  job.  The  work  context 
provides  boundary  conditions  and  the  operating  context  in  which  job  duties  will  be  performed.  It 
can  play  a  critical  role  in  a  person’s  ability  or  willingness  to  handle  job  requirements.  As  noted 
earlier,  RPA  pilots  and  SOs  work  in  a  controlled  environment  where  they  monitor  multiple  video 
screens,  listen  to  input  from  multiple  audio  channels,  and  communicate  with  others  through 
email,  chat,  radios,  and  telephone.  The  work  is  typically  performed  in  shifts  and  may  entail 
rotating,  overnight,  and/or  long  shifts.  Teamwork  is  important,  but  the  core  aircrew  is  small 
during  a  shift  -  a  pilot,  an  SO,  and  an  MC.  Pilots  and  SOs  interact  closely  with  each  other  during 
ifrCTriTrfergBwfmay  interact  wi t tawri y  throuuhw  varietyssfcommunieation 
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channels  (except  for  face-to-face),  ranging  from  the  MC,  to  a  ground-based  JTAC  calling  for 
assistance,  to  an  Anny  or  Marine  officer  whose  troops  are  under  attack,  to  a  high-ranking  officer 
monitoring  the  overall  deployment  of  RPA  resources,  to  non-military  personnel  providing 
intelligence  or  oversight  of  a  mission.  RPA  pilots  and  SOs  may  experience  hours  of  low  activity, 
requiring  self-discipline  to  maintain  vigilance,  interspersed  with  periods  of  intense  activity  often 
characterized  as  “drinking  from  a  fire  hose.”  Operators  are  not  in  any  personal  physical  danger 
when  perfonning  their  duties.  However,  they  do  experience  the  visceral  effects  of  combat,  for 
example,  witnessing  a  friendly  unit  under  attack  by  hostile  forces.  The  duty  station  for  many 
RPA  pilots  and  SOs  is  in  the  continental  U.S.  (CONUS)  and  these  officers  and  airmen  can  return 
home  to  their  families  at  the  end  of  shift.  However,  they  may  volunteer  for  or  be  assigned  to 
temporary  duty  stations  away  from  their  home  base,  or  may  be  deployed  to  other  parts  of  the 
world.  Their  work  is  classified,  so  they  are  not  allowed  to  talk  openly  with  family  or  friends 
about  the  types  of  missions  that  they  fly  or  things  that  occurred  during  a  mission. 

We  decided  to  delineate  the  work  context  factors  found  in  most  RPA  work  environments 
and  that,  as  a  whole,  make  this  career  field  unique.  Our  starting  point  is  shown  in  Appendix  C. 

To  develop  the  list  shown  in  Appendix  C,  we  started  with  information  from  an  existing  Army 
project  that  defined  major  work  context  factors  for  military  jobs  (Army  O 515  NET;  Russell, 
Sinclair,  Erdheim,  Ingerick,  Owens,  Peterson,  &  Pearlman,  2008),  supplemented  by  information 
from  early  conversations  with  SMEs,  information  about  the  RPA  work  context  provided  by  the 
AFPC  project  team,  information  found  in  Bailey  (2008)  and  in  front-end  analysis  reports  (Nagy, 
Eaton,  &  Must,  2006b;  Nagy,  Kalita,  &  Eaton,  2006a),  and  Internet  searches  for  images  and 
popular  press  articles  about  RPAs. 

Work  context  factors  are  likely  to  be  particularly  promising  for  the  development  of  a  new 
screening  measure  because,  unlike  many  of  the  ability  and  skill  constructs  found  in  Tables  2  and 
3,  they  represent  dimensions  that  are  largely  untapped  by  existing  predictors  available  to  the 
USAF  (e.g.,  Armed  Services  Vocational  Aptitude  Battery  [ASVAB],  Air  Force  Officer 
Qualifying  Test  [AFOQT],  and  Test  of  Basic  Aviation  Skills  [TBAS]).  That  is,  the  ASVAB, 
AFOQT,  and  TBAS  provide  good  coverage  of  the  cognitive  and  psychomotor  domains.  The 
Self-Description  Inventory  (SDI+),  which  is  administered  experimentally  as  part  of  the 
AFOQT,  provides  coverage  of  several  non-cognitive  work  style  dimensions.  The  incremental 
validity  over  these  existing  and  established  measures  provided  by  any  newly  developed 
assessment  of  ability  or  work  styles  could  be  small.  Work  context  factors,  as  measured  by  a 
Person-Environment  fit  (P-E  fit)  assessment  or  provided  as  a  realistic  job  preview  (RJP), 
however,  offer  an  opportunity  to  provide  meaningful  enhancements  to  the  selection  system. 

We  received  a  great  deal  of  input  on  the  work  context  factors  from  SMEs.  This  was  a 
topic  they  enjoyed  talking  about  and  one  they  felt  was  important  for  others  to  understand.  The 
final  list  of  key  work  context  factors,  shown  in  Table  4,  was  substantially  different  and  much 
more  customized  to  the  RPA  work  context  than  our  original  list.  Note  that  this  is  not  intended  to 
be  a  comprehensive  description  of  the  RPA  work  environment.  Rather,  it  is  intended  to  capture 
important  aspects  of  the  work  context,  in  particular  the  negative  aspects,  that  incumbents  must 
be  willing  to  tolerate  if  making  RPAs  a  career. 
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Table  4.  Key  Work  Context  Factors  in  the  RPA  Environment 
#  Work  Context  Statement 


Statements  that  are  characteristic  of  the  RPA  work  context 

1  Make  decisions  that  could  have  a  significant  impact  on  the  well-being  of  others. 

2  Work  provides  opportunities  to  protect  friendly  forces  with  minimal  danger  to  self. 

3  A  job  where  one  may  have  to  provide  negative  performance  feedback  to  others. 

4  Perform  work  that  involves  reconnaissance  and  surveillance. 

5  Work  with  others  at  a  distance  or  non  face-to-face  (e.g.,  through  chat,  e-mail,  or  radio). 

6  Required  to  take  action  in  the  face  of  conflicting  or  ambiguous  directions,  orders,  or  priorities. 

7  Work  in  a  setting  where  others  will  observe  or  monitor  what  you  do. 

8  Work  has  a  direct,  immediate,  and  visible  impact  on  mission  success. 

9  Not  allowed  to  talk  about  work  with  family  or  friends. 

10  Remain  alert  and  highly-focused  for  long  periods  of  time,  even  when  there  is  little  happening. 

1 1  Work  as  part  of  a  team  (instead  of  alone)  to  get  tasks  completed. 

12  Work  involves  handling  difficult,  high-stress,  emergency  situations. 

1 3  Work  in  an  occupation  that  is  mentally  challenging. 

14  Take  lethal  action  against  enemy  targets  when  authorized  to  do  so. 

1 5  Work  may  impact  the  success  of  combat  operations  without  risk  of  physical  injury. 

1 6  Perform  many  tasks  simultaneously. 

17  Able  to  establish  roots  in  a  community  due  to  extended  tours  of  duty  in  the  same  location. 

1 8  Work  long  hours  (e.g.,  60-80  hours  per  week). 

1 9  Work  with  cutting  edge  aircraft  technology. 

20  Work  can  quickly  change  from  boring  and  routine  to  very  intense  and  stressful. 

2 1  Work  rotating  duty  shifts. 

22  Persuade  or  influence  others  over  whom  you  have  no  authority  (e.g.,  someone  at  a  higher  rank). 

23  Able  to  experience  flight  without  experiencing  the  physical  challenges  that  accompany  it  (e.g., 
G-forces  or  low  oxygen  levels). 
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Table  4.  (Continued) _ 

#  Work  Context  Statement 


24  Spend  most  of  the  work  shift  in  a  windowless  room,  viewing  multiple  computer  monitors. 

25  Perform  work  under  significant  time  pressure. 

26  Perform  actions  that  could  seriously  injure  or  kill  others  (non-friendly  and  friendly). 

27  Respond  to  frequent  crises  or  emergencies. 

Statements  that  are  NOT  characteristic  of  the  RPA  work  context  (could  be  used  for 
reverse-scored  items  in  a  P-E  fit  assessment) 

28  Able  to  control  the  pace  and  sequencing  of  own  work  activities. 

29  Able  to  set  own  work  schedule. 

30  Work  is  repetitive,  with  very  little  change  in  activities  from  one  day  to  the  next. 

3 1  Perform  work  that  is  dangerous  or  could  result  in  serious  injury  to  self,  (reverse-scored) 

32  Work  requires  physical  hardship  (e.g.,  exposure  to  elements,  sleeping  on  the  ground,  eating 
MRE,  no  toilet) 

33  Afforded  opportunities  to  learn  the  job  from  senior  co-workers  or  supervisors. 

34  Work  allows  room  for  error  and  learning  from  mistakes. 

35  Perform  work  that  does  not  require  memorizing  a  lot  of  details  or  procedures. 

36  Hard  to  know/determine  the  effectiveness  of  your  job  performance. 

37  Perform  work  where  one  can  be  creative. 

38  Work  that  is  easy  to  learn  and  with  few  errors. 

39  Work  alone,  with  little  interaction  with  or  input  from  others. 

40  Work  requires  travel  away  from  home  for  extended  periods  of  time  (e.g.,  several  months). 

41  Perform  work  that  is  physically  demanding. 

42  Work  involves  few  negative  consequences  if  you  make  a  mistake. 

Identifying  Best  Bet  Predictor  Measures 

After  we  finalized  a  list  of  the  most  critical  SAOCs  for  the  RPA  pilot  and  SO  occupations 
and  important  work  context  factors,  our  next  step  was  to  think  about  (a)  potential  measures  of  the 
SAOCs  and  work  context  factors  and  (b)  which  measures  could  be  bundled  into  a  predictor 
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battery  of  reasonable  length  that  would  comprise  measures  appropriate  for  entry-level  selection. 
We  treated  these  as  two  separate  judgments,  first  identifying  best  bet  measures  for  each  critical 
SAOC  and  then  thinking  about  the  most  effective  and  efficient  battery  of  predictor  measures, 
taking  into  account  practical  constraints  on  the  entry-level  selection  process. 

We  started  by  reviewing  available  research  on  RPA  operator  selection  (Bailey,  2008; 
Biggerstaff  et  ah,  1998;  Bruskiewicz,  Houston,  Hezlett,  &  Ferstl,  2007;  Phillips,  2003;  Phillips, 
Arnold,  &  Fatolitis,  2003).  Each  of  these  studies  involve  development  of  entry-level  selection 
tests  for  pilots  of  unmanned  aircraft  only;  not  for  SOs.  None  are  operational  yet.  Only  the  study 
conducted  by  the  Naval  Aerospace  Medical  Research  Laboratory  (NAMRL)  includes  any 
examination  of  the  relationship  between  scores  on  predictor  and  criterion  measures  appropriate 
for  RPA  pilot  performance.  We  attempted  to  locate  information  about  selection  procedures  used 
by  other  agencies  that  fly  RPAs,  such  as  the  Central  Intelligence  Agency  (CIA)  or  Customs  and 
Border  Patrol,  but  could  find  only  general  descriptions  (e.g.,  “must  pass  an  entrance  test  and  be 
able  to  obtain  a  security  clearance”).  Clearly,  there  is  overlap  in  the  duties  and  SAOC 
requirements  for  manned  and  unmanned  aircraft,  so  we  also  reviewed  information  about 
measures  used  for  pilot  selection,  starting  with  two  comprehensive  reviews  conducted  in  the 
past  five  years  (Damos,  2007;  Paullin  et  ah,  2006). 

U.S.  Air  Force  School  of  Aerospace  Medicine  (USAFSAM)  Medical  Flight  Screening 

for  RPA  Pilots  and  SOs 

The  U.S.  Air  Force  School  of  Aerospace  Medicine  (USAFSAM)  administers  medical 
flight  screening  before  candidates  enter  RPA  pilot  or  SO  training.  The  exam  includes 
standardized  and  clinical  evaluations  of  cognitive  functioning,  personality,  psychopathology,  and 
medical  fitness  (e.g.,  vision,  hearing,  and  anthropometric  measures).  It  measures  a  number  of  the 
SAOCs  listed  in  Tables  2  and  3,  plus  medical  and  neuropsychological  factors  that  were  not 
considered  in  the  present  study.  This  screening  process  is  used  to  detect  physical  or 
psychological  issues  that  could  prevent  a  candidate  from  completing  training  or  performing 
effectively  post-training.  In  other  words,  this  is  a  screen  out  process. 

The  focus  of  the  present  project  was  to  identify  measures  that  could  be  administered 
during  the  recruitment  or  accession  process,  or  soon  after  accession,  with  the  goal  of  expanding 
the  number  of  eligible  airmen  and  officers  in  the  RPA  training  pipeline.  In  other  words,  these 
measures  would  be  part  of  a  select  in  process.  Of  course,  any  such  candidates  would  still  need  to 
meet  all  the  requirements  for  entering  RPA  training,  including  passing  the  USAFSAM  medical 
flight  screening. 


Royal  Air  Force  (RAF)  Predator  Battery 

Bailey  (2008)  describes  efforts  to  develop  and  validate  a  selection  battery  for  UK  RAF 
Predator  Pilots.  The  battery  includes  subtests  designed  to  assess  dynamic  processing  of  visual  and 
auditory  information,  including  building  and  maintaining  situational  awareness.  It  attempts  to 
replicate  the  kind  of  cognitive  processing  tasks  that  must  be  perfonned  while  flying  a  Predator. 
This  battery  does  not  purport  to  tap  temperament  or  P-E  fit  requirements. 
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U.S.  Navy  Unmanned  Aerial  Vehicle  (UA  V)  Selection  Battery 

In  the  mid- 1 990 ’s,  NAMRL  was  tasked  to  develop  a  performance-based  selection  battery 
for  UAV  operators.  Biggerstaff  et  al.  (1998)  describe  development  and  initial  validation  of  the 
Computer-Based  Performance  Test  (CBPT).  This  battery  included  psychomotor  tracking  tasks,  a 
dichotic  listening  test,  a  measure  of  mental  rotation  (Manikin  test),  and  a  measure  of  perceptual 
speed  (Digit  Cancellation  test).  Various  subtests  are  administered  concurrently  to  assess  multi¬ 
tasking  (or  time  sharing)  ability.  These  researchers  conducted  a  very  small-scale  validation  study 
based  on  8  UAV  pilot  trainees  and  reported  that  the  CBPT  showed  good  promise  for  predicting 
training  performance.  According  to  Phillips  (2003),  further  research  stalled  due  to  lack  of  funding, 
but  was  able  to  locate  scores  for  39  UAV  pilot  trainees.  Phillips  (2003)  called  the  battery  the 
Psychomotor  Test  (PMT)  but  listed  several  of  the  same  subtests  as  described  by  Biggerstaff  et  al. 
(1998)  as  part  of  the  CBPT.  Phillips  (2003)  and  Phillips,  Arnold,  and  Fatolitis  (2003)  reported 
strong  correlations  between  PMT  scores  for  39  UAV  student  pilots  and  measures  of  training 
performance  (mid  AO’s  to  mid-.50’s)  and  significant  mean  score  differences  between  students 
who  did  and  did  not  complete  the  UAV  operator  training  program.  Phillips  (2003)  recommended 
that  the  Navy  implement  a  UAV  selection  battery  based  on  the  PMT  after  upgrading  the 
programming  or  to  conduct  the  necessary  research  to  validate  and  implement  another  prototype 
performance-based  measure  that  also  was  developed  by  NAMRL  called  the  Automated  Spatial 
and  Cognitive  Abilities  Test  (ASCAT).  According  to  Phillips  (2003),  ASCAT  includes  several 
subtests  that  are  the  same  as  or  similar  to  the  PMT  subtests,  but  was  written  in  a  modem 
programming  language  and  could  be  web-enabled.  One  of  the  subtests  included  in  ASCAT 
appears  very  similar  to  the  Direction  Orientation  Test  that  also  appears  in  the  USAF  Test  of  Basic 
Aviation  Skills  (TBAS).  We  could  not  locate  any  information  about  the  current  status  of  the 
CBPT,  PMT,  or  ASCAT  selection  batteries. 

U.S.  Army  Entry-level  Selection  Battery  for  RPA  Operators 

Bruskiewicz  et  al.  (2007)  recommended  a  selection  battery  for  RPA  operators  based  on  a 
taxonomic  approach  similar  to  the  one  taken  in  this  project.  To  our  knowledge,  this  system  is  not 
in  operational  use  (personal  communication,  Lawrence  Katz,  October  10,  2009).  The  research 
team  started  with  predictor  constructs  identified  as  relevant  for  Army  RPA  operators  and  then 
identified  a  best  bet  selection  battery,  choosing  from  measures  that  already  exist.  Most  of  the 
recommended  measures  are  taken  from  a  selection  battery  that  the  same  research  team  developed 
for  Army  rotary-wing  aviators  (Army  Selection  Instrument  for  Flight  Training  [SIFT];  Houston 
&  Bruskiewicz,  2006)  and  which  incorporates  several  cognitive  subtests  drawn  from  the  U.S. 
Navy’s  Aviator  Selection  Test  Battery  (ASTB).  The  recommended  battery  taps  crystalized 
intelligence  (e.g.,  Reading  Comprehension),  fluid  intelligence  (e.g.,  Spatial  Apperception),  and 
perceptual  speed  and  accuracy,  and  also  includes  measures  of  temperament  variables  (self-report 
biodata  and  temperament  scales). 


Pilot  Selection  Batteries 

We  also  examined  pilot  selection  batteries  currently  in  use  by  the  U.S.  military,  or  ones 
not  currently  in  use,  but  may  hold  some  promise  for  RPA  selection.  These  selection  batteries 
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clearly  measure  several  of  the  SAOCs  identified  as  critical  for  RPA  pilots  and  SOs.  Summary 
information  is  provided  in  Appendix  B  of  this  report,  and  several  of  these  batteries  are  described 
in  more  detail  in  Paullin  et  al.  (2006)  and  Houston  and  Bruskiewicz  (2006).  Briefly,  the  USAF 
uses  scores  on  specific  AFOQT  subtests  combined  with  scores  on  the  TBAS  (Carretta,  2005)  as 
part  of  its  pilot  trainee  selection  process.  The  U.S.  Navy  uses  scores  on  the  ASTB.  The  U.S. 
Army  developed  the  SIFT  for  selection  of  rotary  wing  aviators  in  the  mid-2000’s  (Houston  & 
Bruskiewicz,  2006). 


Other  Measures  that  Tap  Relevant  SAOCs 

We  located  several  other  test  batteries  or  stand-alone  measures  that  tap  SAOCs  important 
for  RPA  pilots  and  SOs.  For  enlisted  personnel,  one  of  the  primary  measures  is  the  ASVAB, 
which  every  enlisted  applicant  must  complete  at  the  time  of  enlistment.  Many  of  the  existing  test 
batteries,  including  the  ASVAB,  contain  several  measures  of  crystalized  intelligence  (Carroll, 
1993;  Horn,  1965;  Horn  &  Noll,  1997),  such  as  verbal  and  numerical  reasoning  and  reading 
comprehension  and  vocabulary.  Increasingly,  entry-level  test  batteries  also  include  measures  of 
fluid  intelligence,  such  as  measures  of  spatial  ability,  nonverbal  or  figural  reasoning,  and 
working  memory. 

Several  military  research  efforts  have  focused  on  developing  measures  to  tap 
psychomotor  and  cognitive  processing  constructs.  Examples  include  the  USAF  Basic  Attributes 
Test  (BAT)  (Carretta,  1987),  which  was  replaced  by  TBAS  (Carretta,  2005),  and  a  joint  effort  by 
the  U.S.  Anny  and  the  U.S.  Navy  to  develop  the  Enhanced  Computer- Administered  Test 
(ECAT)  battery  that  incorporates  measures  of  perceptual  speed  and  accuracy,  spatial  and 
inductive  reasoning,  and  psychomotor  skills  (Alderton,  Wolfe,  &  Larson,  1997;  Russell, 

Peterson,  Rosse,  Hatten,  McHenry,  &  Houston,  2001).  Embretson  (1998)  developed  the  Abstract 
Reasoning  Test  (ART)  which  demonstrated  good  construct  validity  when  administered  to 
samples  of  USAF  recruits.  Currently,  the  U.S.  Department  of  Defense  is  considering  including 
the  ART  or  the  ECAT  figural  reasoning  test  in  the  ASVAB  (Drasgow,  Embretson,  Kyllonen,  & 
Schmitt,  2006). 

Ackerman  and  colleagues  conducted  a  series  of  research  studies  to  explore  the  domains 
of  psychomotor  skills  and  perceptual  speed.  They  developed  PC-based  measures  of  both 
psychomotor  and  perceptual  speed,  taking  advantage  of  advances  in  computer  software  and 
hardware  (e.g.,  touch  screens)  but  also  using  standard  peripheral  equipment  that  should  be 
readily  available  (e.g.,  light  pens)  (Ackerman,  2004;  2007;  Ackerman,  Beier,  &  Boyle,  2002; 
Ackerman  &  Cianciolo,  2000;  Ackerman  &  Beier,  2007). 

The  U.S.  Navy  has  also  developed  performance-based  measures  that,  to  date,  have  only 
been  used  for  experimental  research,  including  a  block  rotation  task  and  a  two-dimensional 
compensatory  tracking  task  (Fatolitis,  Jentsch,  Hancock,  Kennedy,  &  Bowers,  2010).  It  has  also 
used  a  measure,  called  SYNWORK  in  early  versions  (Elsmore,  1994)  and  SYNWIN  in  later 
versions  (Oswald,  Hambrick,  Jones,  &  Ghumman,  2007),  to  measure  multi-tasking  performance. 
To  date,  this  multi-tasking  measure  has  been  used  as  a  criterion  against  which  other  predictor 
measures  were  validated  (  Branscome  &  Grynovicki,  2007;  Oswald  et  al.,  2007),  but  we 
examined  it  as  a  possible  predictor  measure. 
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The  Federal  Aviation  Administration’s  (FAA)  Air  Traffic  Selection  and  Training 
(ATSAT)  battery  measures  skills  and  abilities  necessary  to  process  dynamic  information 
presented  on  two-dimensional  displays,  with  a  strong  component  of  spatial  orientation  and 
visualization,  while  at  the  same  time  communicating  through  audio  channels  with  external 
parties  (King,  Manning,  &  Drechsler,  2007;  Ramos,  Heil,  &  Manning,  2001a,  2001b;  Wise, 
Tsacoumis,  Waugh,  &  Putka,  2003;  Wise,  Tsacoumis,  Waugh,  Putka,  &  Horn,  2001).  Several  of 
the  subtests  within  this  battery  measure  SAOCs  required  for  RPA  pilots  and  SOs.  Carretta  and 
King  (2008)  administered  the  ATSAT  to  USAF  enlisted  personnel  who  were  training  to  become 
air  traffic  controllers  (ATC).  They  found  that  an  ATSAT  subtest  that  simulates  some  aspects  of 
the  ATC  job  provided  incremental  validity  beyond  the  ASVAB  in  predicting  ATC  training 
outcomes.  However,  they  were  hesitant  to  recommend  that  the  USAF  begin  administering  this 
subtest  operationally  because  it  required  95  minutes  of  testing  time  and,  when  combined  with  the 
ASVAB,  increased  mean  subgroup  score  differences. 

In  the  non-cognitive  arena,  there  are  existing  measures  of  several  different  types  of 
constructs  including: 

•  personality  and  temperament  constructs,  for  example,  the  USAF  SDI+  (Metrica, 

1997;  Weissmuller  &  Schwartz,  2008),  the  U.S.  Army  Rational  Biodata  Inventory 
(RBI;  Kilcullen,  Mael,  Goodwin,  &  Zazanis,  1999;  Kilcullen,  White,  Sanders,  & 
Hazlett,  2003),  and  the  U.S.  Army  Assessment  of  Individual  Motivation  (AIM; 

White  &  Young,  1998;  White,  Young,  &  Rumsey,  2001) 

•  vocational  interests,  for  example,  the  USAF  Vocational  Interest-Career  Exploration 
(VOICE;  Alley  &  Matthews,  1982).  the  U.S.  Navy  Job  Opportunities  in  the  Navy 
(JOIN;  Alley,  2000;  Chen  &  Jones,  2008;  Farmer,  Bearden,  Fedak,  et  ah,  2006),  or 
the  U.S.  Anny  Work  Preferences  Assessment  (Van  Iddekinge,  Putka,  &  Sager, 

2005),  and 

•  values,  for  example,  the  U.S.  Army  Work  Values  Inventory  (Van  Iddekinge,  Putka, 

&  Sager,  2005). 

Several  service  branches  are  currently  exploring  new  approaches  to  measuring  non- 
cognitive  characteristics  in  an  attempt  to  overcome  faking  concerns  associated  with  self-report 
instruments,  for  example,  the  Navy  Computer  Adaptive  Personality  Scales  (NCAPS;  Borman, 
Schneider,  &  Houston,  2009;  Houston,  Borman,  Fanner,  &  Bearden,  2006)  and  the  Army 
Tailored  Adaptive  Personality  Assessment  System  (TAPAS;  Heffner,  White,  &  Owens,  2010; 
Stark,  Chernyshenko,  &  Drasgow,  2010).  The  SDI+,  NCAPS,  and  TAPAS  are  measures  of  the 
Big  Five  personality  constructs. 


Best  Measure(s)  for  Each  SAOC 

As  expected,  we  found  existing,  proven  measures  for  many  of  the  most  critical  SAOCs, 
including  several  that  the  Air  Force  already  owns  and  uses.  Appendix  B  summarizes  information 
about  several  such  measures,  some  of  them  bundled  into  test  batteries  and  some  of  them 
measures  of  a  single  SAOC.  We  also  created  an  SAOC  by  predictor  matrix,  shown  in  Table  5, 
which  includes  a  “best  bet”  existing  predictor  as  well  as  one  or  more  alternate  predictors  for  each 
SAOC-where  we  were  able  to  find  them.  We  focused  first  on  identifying  measures  already  in  use  by 
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the  Air  Force.  For  example,  the  AFOQT  and  SDI+  are  administered  in  the  USAF  officer 
commissioning  process  and  the  ASVAB  is  administered  in  the  enlistment  qualification  process.  These 
two  multi-  scale  test  batteries  provide  coverage  of  several  critical  SAOCs  (although  faking  could  be  a 
concern  when  the  SDI+  is  used  for  high-stakes  selection  purposes).  The  USAF  already  administers 
the  TBAS  as  part  of  the  screening  process  for  officers  who  want  to  become  pilots  and  it  has  also  been 
administered  experimentally  to  enlisted  personnel.  We  also  identified  several  alternate  measures  that 
should  be  available  to  the  USAF,  either  because  they  were  developed  with  Air  Force  research  funding 
(e.g.,  PC-based  measures  of  psychomotor  skills  and  perceptual  speed  [Ackerman,  2004])  or  because 
they  are  owned  by  other  U.S.  military  service  branches  (e.g.,  the  U.S.  Army  SIFT  or  the  U.S.  Navy 
ASTB). 


We  did  not  include  the  UK  RAF  Predator  battery  in  Table  5  because  it  was  not  complete  at 
the  time  of  this  project  (Bailey,  2008).  When  the  UK  RAF  Predator  battery  is  complete,  it  may 
provide  a  viable  option,  though  we  believe  it  would  need  to  be  supplemented  with  some  measures 
of  non-  cognitive  SAOCs.  Similarly,  we  excluded  the  measures  that  are  part  of  the  medical  flight 
screening  battery  used  by  USAFSAM.  We  are  not  trying  to  replace  or  pre-empt  the  medical  flight 
screening  process,  so  we  did  not  want  to  build  any  redundancy  into  the  selection  process.  We 
included  the  ECAT  measures  but  note  that  the  programming  for  at  least  some  of  these  tests  would 
need  to  be  updated  before  they  could  be  used  operationally.  We  also  included  the  NCAPS  scales  as 
alternates  for  several  of  the  non-cognitive  SAOCs,  but  note  that  they  have  not  yet  been  fully 
approved  by  the  Navy  for  operational  use.  Finally,  we  included  several  TAPAS  scales  but  note  that 
the  number  of  TAPAS  scales  and  their  labels  have  changed  over  time  and  will  likely  continue  to  do 
so. 


If  the  USAF  decides  to  pursue  an  RPA-specific  entry-level  screening  system,  it  will  need  to 
consider  the  feasibility  of  administering  a  computer-based  assessment  that  includes  special 
equipment  (e.g.,  joystick  and  rudder  pedals  required  for  TBAS)  or  the  feasibility  of  administering  a 
test  currently  used  for  one  population  (e.g.,  officers)  in  a  different  population  (e.g.,  enlisted). 

Measurement  Gaps 

As  shown  in  Table  5,  the  critical  SAOCs  for  which  there  is  a  clear  measurement  gap 
include  Oral  Expression  Skills  and  Judgment  and  Decision  Making.  In  addition,  there  is  no 
existing  P-E  fit  measure  customized  to  the  RPA  work  context.  For  each  of  the  other  SAOCs, 
there  is  an  existing  measure  that  provides  at  least  a  reasonable  degree  of  coverage.  However, 
some  of  the  measures  are  less  than  a  perfect  fit.  The  gap  analysis  assumes  that  it  would  be 
feasible  to  administer  any  of  the  measures  to  both  an  enlisted  and  an  officer  population,  however 
this  assumption  may  be  a  stretch.  For  example,  it  may  or  may  not  be  possible  to  administer 
TBAS  to  all  enlisted  personnel.  Another  potential  issue  with  TBAS  is  that  one  of  the  two 
tracking  tasks  involves  rudder  pedals.  In  the  RPA  context,  only  the  pilot  uses  rudder  pedals,  so  it 
may  be  necessary  to  create  a  second  version  of  this  tracking  task  that  involves  a  trackball  rather 
than  rudder  pedals  (because  SOs  use  a  trackball).  Finally,  TBAS  provides  a  measure  of  Time 
Sharing  ability,  but  does  so  by  combining  psychomotor  tasks  with  cognitive  processing  tasks.  A 
purer  measure  of  Time  Sharing  ability  would  combine  only  cognitive  processing  tasks.  Beyond 
TBAS,  some  of  the  best  bet  assessments  are  commercially- available  instruments  for  which  there 
would  be  licensing  or  usage  fees,  and  we  do  not  know  if  the  Air  Force  would  be  willing  to  pay 
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such  fees  for  large-scale  testing  programs.  Where  the  only  option  was  a  commercially- 
available  instrument,  we  used  the  label  “Maybe”  in  the  gap  column. 


Table  5.  Best  Bet  Existing  Measures  for  Each  Critical  SAOC 


Type 

SAOC 

Best  Bet  Existing  Air 
Force-owned  Predictor 

Alternate  Predictor(s)  that  the  Air 
Force  Could  Fikely  Access 

Gap? 

Non-Cog 

Initiative 

SDI+  Conscientiousness 
Factor  score  (officer) 

TAPAS  Achievement  scale 

NCAPS  Achievement  Striving  &  Self- 
Reliance  scales 

No 

Non-Cog 

Assertiveness 

SDI+  Extroversion  Factor 
score  (officer) 

TAPAS  Dominance  Scale 

NCAPS  Leadership  Orientation 

No 

Non-Cog 

Decisiveness 

SDI+  Extroversion  Factor 
score  (officer) 

TAPAS  Dominance  Scale 

NCAPS  Leadership  Orientation 

No 

Non-Cog 

Self  Control 

SDI+  Emotional  Stability 
Factor  score  (officer) 

TAPAS  Even-Tempered  scale 

NCAPS  Self-Control/Impulsivity 

No 

Non-Cog 

Stress 

Tolerance 

SDI+  Emotional  Stability 
Factor  score  (officer) 

TAPAS  Adjustment  scale 

NCAPS  Stress  Tolerance  scale 

No 

Non-Cog 

Adaptability 

SDI+  Emotional  Stability 
Factor  score  (officer) 

NCAPS  Adaptability/Flexibility  scale 

Army  Adaptability  Measures 

No 

Ability 

Working 

Memory 

TBAS  Directed  Listening 
test 

ECAT  Mental  Counters  or  Sequential 
Memory  test 

No 

Ability 

Task 

Prioritization 

TBAS  Emergency 

Scenarios 

ATSAT  Letter  Factory  or  Air  Traffic 
Scenarios  tests 

No 

Ability 

Selective 

Attention 

TBAS  Emergency 

Scenarios  to  a  limited  extent 

ATSAT  Letter  Factory  or  Air  Traffic 
Scenarios  tests 

No 

Ability 

Time  Sharing 

TBAS  Directed  Listening  + 
Tracking  Tasks 

ATSAT  Letter  Factory  or  Air  Traffic 
Scenarios  tests;  SYNWIN 

No 

Ability 

Perceptual 

Speed 

AFOQT  Table  Reading  test 
(officer) 

Army  SIFT  battery:  PS&A  Simple 
Drawings  subtest; 

PC -based  measure  of  Perceptual  Speed; 
ATSAT  Dials  Test 

No 

Ability 

Spatial 

Orientation 

TBAS  Direction  Orientation 

Test 

ECAT  Spatial  Orientation;  ASTB  Spatial 
Apperception 

No 

Ability 

Visualization 

AFOQT  Block  Counting  & 
Rotated  Blocks  (officer) 
ASVAB  Assembling 

Objects  (enlisted) 

NAMRL  Block  Rotation  Test  or  Manikin 
Tests 

No 
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Table  5.  (Continued) 

Type 

SAOC 

Best  Bet  Existing  Air 
Force-owned  Predictor 

Alternate  Predictor(s)  that  the  Air 
Force  Could  Likely  Access 

Gap? 

Ability 

Pattern 

Recognition 

AFOQT  Flidden  Figures  test 
(officer) 

ASVAB  Assembling 

Objects  (enlisted)  [gets  at 
the  higher-order  spatial 
ability  factor] 

PC-based  measure  of  Perceptual  Speed- 
Pattern  Recognition 

No 

Ability 

Oral 

Comprehension 

TBAS  Directed  Listening 
test  (weak  measure) 

None 

No 

Ability 

Oral 

Expression 

None 

None 

Yes 

Ability 

Number 

Facility 

AFOQT  Arithmetic 
Reasoning  (officer) 

ASVAB  Arithmetic 
Reasoning  (enlisted) 

ASTB  Mathematical  Ability; 

PC -based  measure  of  Perceptual  Speed  - 
Factors  of  7 

No 

(These  measures  do  not 
necessarily  capture  the 
dynamic  way  in  which  this 
ability  is  used  in  the  RPA 
work  context) 

Ability 

Control 

Precision 

TBAS  Vertical  Tracking 
Test(VTT)&  Airplane 
(Horizontal  Tracking) 
subtests 

EC  AT  One-Hand  &  Two-Hand  Tracking; 
NAMRL  ASCAT  Control  Reversal  Test; 

Mirror  or  Maze  Tracing  PC-based 
psychomotor  tests,  but  would  need 
research  to  determine  if  the  response 
modality  (light  pen  or  touch  screen)  is 
similar  enough  to  tracking  tools  used  in 
RPA  Ground  Control  Stations 

No 

Skill 

Critical 

Thinking 

None 

Watson-Glaser  Critical  Thinking 

Appraisal 

Maybe 

Skill 

Judgment  and 
Decision 

Making 

None 

None 

Yes 

Skill 

T  eamwork 

Skills 

SDI+  Team  Scale  (officer) 

Teamwork  Knowledge,  Skills  and  Ability 
Test 

No 

P-E  Fit 

RPA  Work 
Context 

Features 

None 

None 

Yes 
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We  presented  our  preliminary  recommendations  to  the  AFPC  project  team.  They  were 
already  very  knowledgeable  about  most  of  the  measures  listed  in  Table  5.  Our  discussion  focused 
on  (a)  potential  measurement  gaps,  (b)  identifying  SAOCs  that  would  most  appropriately  be 
measured  in  an  entry-level  selection  process,  and  (c)  practical  constraints  on  the  number  and  type 
of  assessments  that  could  be  measured.  After  discussion,  we  reached  consensus  that  several  of 
the  critical  SAOCs  need  not  or  should  not  be  given  further  consideration  at  this  time.  Our 
rationale  is  provided  below.  Finally,  we  decided  to  consult  with  RPA  SMEs  on  several  existing 
measures  of  Time-Sharing  ability,  to  collect  their  views  of  the  extent  to  which  these  measures 
adequately  capture  critical  aspects  of  Time  Sharing  as  it  occurs  in  the  RPA  context. 

Oral  Comprehension  and  Oral  Expression  Skills 

SMEs  told  us  RPA  pilots  and  SOs  must  be  able  to  quickly  comprehend  call  signs  and 
other  cryptic  messages  that  may  be  arriving  through  the  audio  channel,  while  they  are  also 
devoting  some  of  their  attention  to  other  tasks.  SMEs  also  indicated  that  the  most  critical  aspect 
of  oral  expression  is  learning  and  accurately  using  the  shorthand  speech  required  to  communicate 
efficiently  and  effectively  with  their  aircrew  partners  and  with  outside  entities.  Many  SMEs 
indicated  that  “communication  skills”  are  especially  important  for  RPA  SOs,  but  further  probing 
revealed  that  the  critical  SAOC  is  not  communication  skills,  per  se,  but  having  the  assertiveness 
to  speak  up  and  question  the  pilot  (who  is  an  officer),  for  example,  when  he/she  seems  on  the 
verge  of  making  an  error.  Learning  to  comprehend  call  signs  and  to  speak  in  the  type  of 
shorthand  required  by  the  RPA  job  is  something  that  can  be  learned  after  hire.  Being  willing  to 
speak  up  to  a  higher-ranking  person  is  captured  by  the  Assertiveness  SAOC. 

We  spent  some  time  searching  for  existing  measures  of  communication  skills,  and  these 
are  summarized  in  Appendix  B.  We  found  that  most  are  designed  for  business-like  settings  in 
which  oral  communication  skills  are  important  for  doing  briefings  or  sales  presentations  or 
delivering  a  training  session.  None  emulated  the  type  of  oral  comprehension  and  oral  expression 
skills  that  are  unique  to  the  RPA  work  environment. 

Critical  Thinking  Skills 

We  spent  time  considering  this  skill  because  it  seems  important  for  all  Air  Force  officers 
and  enlisted  personnel,  including  RPA  pilots  and  SOs.  Watson  and  Glaser  (1994)  were  the  first 
to  define  a  concept  they  labeled  “critical  thinking  skills”  and  developed  a  measure  that  is  still  in 
use  today  (Watson  &  Glaser,  2009).  They  define  critical  thinking  as  attitudes,  knowledge,  and 
skills  that,  together,  are  required  to  understand  (a)  that  factual  statements  or  arguments  must  be 
supported  by  some  kind  of  evidence  and  (b)  how  to  evaluate  the  relevance,  accuracy,  and 
reliability  of  such  evidence.  Scores  on  the  Watson-Glaser  Critical  Thinking  Appraisal  II 
(Watson  &  Glaser,  2009)  are  correlated  in  the  .45-. 65  range  with  other  measures  of  cognitive 
ability  and  in  the  .25-. 35  range  with  some  measures  of  Openness  to  Experience.  A  few  other  test 
publishers  provide  similar  measures.  In  a  series  of  studies  conducted  by  the  U.S.  Army  Research 
Institutes  for  the  Behavioral  and  Social  Sciences  (ARI),  critical  thinking  was  treated  as  a  skill 
that  could  be  trained,  and  researchers  developed  a  tool  called  the  Computerized  Training  in 
Critical  Thinking  (CT2)  (Fischer,  Spiker,  Harris,  McPeters,  &  Riedel,  2008).  This  training 
approach  is  consistent  with  calls  by  education  experts  and  other  commentators  for  stronger  and 
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more  widespread  training  in  critical  thinking  skills  for  both  children  and  adults  (e.g.,  Brookfield, 
1987;  Facione,  Facione,  &  Giancarlo,  2000;  Sternberg,  1986).  It  appears  that  critical  thinking 
skills  are  somewhat  domain-specific  (the  knowledge  component  of  the  definition)  and  they  may 
also  be  trainable.  As  trainable  skills,  they  need  not  be  a  high-priority  target  for  entry-level 
selection.  Finally,  we  were  concerned  that  measures  of  critical  thinking  skills  might  not  provide 
enough  incremental  validity  beyond  measures  of  fundamental  cognitive  abilities  to  be  worth  the 
extra  administration  time. 

We  located  several  potential  measures  of  critical  thinking  skills,  as  shown  in  Appendix  B. 
For  the  reasons  just  cited,  we  did  not  recommend  this  type  of  measure  for  further  consideration. 

Judgment  and  Decision  Making  Skills 

Judgment  and  decision  making  skills  are  typically  measured  through  structured  interview 
questions  or  through  scenario-based  assessments  that  provide  some  context  in  which  judgment 
must  be  applied  and  decisions  made.  Situational  judgment  tests  (SJTs)  have  been  used  for  this 
purpose  (Weekly  &  Ployhart,  2006)  as  have  live  or  virtual  role  play  and  in-basket  exercises 
(Thornton  &  Rupp,  2004;  Tsacoumis,  2007).  A  customized,  computer-administered  SJT  or 
virtual  role  play  could  be  developed  to  measure  general  judgment  and  decision  making  skills,  and 
these  measures  could  be  customized  to  be  face  valid  for  the  RPA  work  context.  In  the  RPA  work 
context,  decisions  often  must  be  made  under  time  pressure  and  there  may  not  be  time  to 
formulate  and  evaluate  several  alternatives,  as  in  the  classic  sense  of  judgment  and  decision 
making.  A  customized  SJT  or  virtual  role  play  could  emulate  at  least  some  of  the  speeded  aspects 
of  judgment  and  decision  making  as  it  occurs  in  the  RPA  work  context.  We  did  not  find  any 
viable  measure  of  speeded,  contextualized  judgment  and  decision  making,  so  we  did  not  pursue 
this  gap  further,  but  note  that  it  could  be  an  area  for  further  research. 

Teamwork  Skills 

There  are  stable  individual  differences  in  preferences  for  working  in  teams  versus  alone, 
and  in  levels  of  agreeableness  and  cooperativeness  that  facilitate  team  processes.  These  qualities 
can  be  measured  with  existing  tools,  such  as  the  SDI+,  TAPAS,  or  NCAPS.  Stevens  and 
Campion  (1999)  developed  and  validated  a  scenario-based  test  of  teamwork  knowledge,  skills, 
and  abilities  (KSAs)  that  has  proven  predictive  of  team-oriented  job  performance  in  their  own 
research  and  in  that  of  others  (McClough  &  Rogelberg,  2003).  A  promising  aspect  of  the 
Teamwork  KSA  Test  is  that  it  should  be  less  vulnerable  to  faking  than  self-report  inventories 
that  tap  preferences  and  attitudes.  Therefore,  the  Teamwork  KSA  test  provides  a  viable 
alternative  if  concerns  about  faking  limit  the  use  of  self-report  measures  of  teamwork 
preferences  and  attitudes.  However,  it  is  also  possible  that  this  test  would  overlap  considerably 
with  existing  measures  of  cognitive  ability  or  teamwork  preferences  and  attitudes.  Stevens  and 
Campion  (1999)  found  incremental  validity  for  the  Teamwork  KSA  test  beyond  that  shown  by 
an  employment  aptitude  battery  in  one  study,  but  not  in  another  study.  Given  that  the  test  would 
require  more  than  30  minutes  of  testing  time  and  concerns  that  it  might  not  add  any  predictive 
value,  we  decided  not  to  consider  this  measure  further  at  this  time. 
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Time  Sharing 

As  shown  in  Table  5,  we  identified  several  existing  measures  of  Time  Sharing  that  should 
be  accessible  to  the  Air  Force:  (a)  TBAS  combinations  of  tracking  and  listening  tasks,  (b) 
ATSAT  Letter  Factory  Test,  (c)  ATSAT  Air  Traffic  Scenarios  Test,  and  (d)  SYNWIN.  We  did 
not  include  the  NAMRL  tests  because  we  could  not  get  a  clear  sense  for  the  current  status  of 
these  measures,  for  example,  whether  or  not  they  have  been  programmed  in  modem 
programming  languages.  In  August  2010,  we  conducted  focus  groups  with  several  RPA  SMEs, 
in  which  we  provided  descriptions  and  screen  shots  of  the  various  measures.  (It  was  not  possible 
to  provide  live  demonstrations.)  The  SMEs  included  four  experienced  ANG  RPA  pilots,  four 
experienced  ANG  RPA  SOs,  and  one  contractor  who  has  flown  RPAs  and  has  trained  RP  A 
pilots.  The  SMEs  felt  some  aspects  of  each  measure  reflected  RPA  time  sharing  requirements, 
while  other  aspects  did  not.  Their  comments  are  summarized  in  Figure  3. 


Things  TBAS  gets  right: 

•  Doing  more  than  one  thing  at  once 

•  Audio  coupled  with  vertical  tracking  is  somewhat  realistic  (though  not  tracking  another 
aircraft);  occasionally  having  to  deal  with  an  emergency 

Things  TBAS  does  not  get  right: 

•  Horizontal  tracking  is  only  relevant  for  RPA  pilots  who  do  launch  and  recovery. 

•  Stick-and-rudder  skills  are  not  the  most  critical  aspects  of  multi-tasking,  particularly  for 
pilots  who  are  not  involved  in  launch  and  recovery.  This  test  emphasizes  those  skills. 

•  Tracking  other  aircraft  in  crosshairs 

Things  the  ATSAT  Letter  Factory  Test  gets  right: 

•  Continual  monitoring  of  things  that  are  changing/moving 

•  Must  keep  ahead  of  the  system,  by  being  required  to  order  boxes 

•  Quality  control,  noticing  a  problem  and  taking  action 

•  Situational  awareness  questions,  particularly  if  they  get  at  spatial  awareness  of  what  was 
going  on  in  the  workspace  when  it  disappeared 

•  Task  saturation  can  increase  with  change  in  pace  of  belts  or  number  of  errors,  etc. 

Things  the  ATSAT  Letter  Factory  Test  does  not  get  right: 

•  No  audio  component 

•  No  altitude  or  3ld  dimension  component 

•  No  tracking  component 

Things  the  ATSAT  Air  Traffic  Scenarios  Test  gets  right: 

•  Requirement  to  understand  and  deal  with  headings,  altitude,  and  speed 

•  Requirement  to  think  in  3  dimensions 

•  Prioritization  is  embedded  in  the  external  environment  (e.g.,  have  to  quickly  determine 
which  plane  must  have  attention  right  now,  which  ones  can  wait  a  little  while) 

•  Requirement  to  monitor  aircraft  to  ensure  instructions  were  received  accurately 

•  Need  to  “stay  ahead” _ 
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•  Multiple  aircraft  in  the  same  space  moving  at  different  speeds;  must  perform  tasks  both 
quickly  and  accurately 

•  Screen  display  is  a  little  like  FalconView 

•  Monitoring  position  (distance,  heading,  and  altitude)  of  other  aircraft,  as  well  as  their  speed 

•  Monitoring  something  that  is  dynamic  and  constantly  changing 

Things  the  ATSAT  Air  Traffic  Scenarios  Test  does  not  get  right: 

•  No  audio  component 

•  No  tracking  (as  with  a  joystick  or  trackball)  component 

•  Landing  fields  are  not  accurate,  but  concept  that  there  is  something  stationary  on  the 
ground  is  okay 

Things  the  SYNWIN  test  gets  right: 

•  Several  things  happening  at  once,  with  a  mix  of  monitoring  and  actively  doing  something 
(e.g.,  solve  math  problems) 

•  Audio  component,  with  only  some  of  the  information  relevant 

•  Better  if  you  can  perform  tasks  both  quickly  and  accurately 

•  Monitoring  fuel  gauge  is  face  valid 

•  Simulates  checking  on  operations  while  handling  emergency  procedures  and  getting  audio 
messages 

•  Constantly  scanning,  some  activities  require  more  attention  than  others,  some  are  more 
active  than  others 

•  Math  part  simulates  calculating  a  fuel  plan 

•  Audio  component,  with  only  some  of  the  information  relevant 

•  Shows  that  you  can  do  other  stuff  all  the  time 

Things  the  SYNWIN  test  does  not  get  right: 

•  No  psychomotor  tracking  component 

•  No  clear  requirement  to  prioritize  tasks _ 

Figure  3.  Summary  of  RPA  SME  comments  on  measures  of  time  sharing  ability. 

We  also  considered  the  length  of  each  of  the  preceding  measures  of  Time  Sharing  ability, 
its  face  validity,  and  its  likely  availability  to  the  USAF.  Given  that  TBAS  is  already  administered 
to  officer  pilot  candidates  and  to  at  least  some  enlisted  personnel,  we  recommend  continuing  to 
administer  it.  However,  we  note  that  the  tracking  task  that  uses  rudder  pedals  is  not  an  accurate 
representation  of  psychomotor  skills  used  by  RPA  SOs.  The  ATSAT  Air  Traffic  Scenarios 
subtest  is  lengthy,  requiring  more  than  90  minutes  to  administer.  Thus,  it  does  not  seem  viable  for 
inclusion  in  an  entry-level  selection  process.  The  ATSAT  Letter  Factory  Test  is  also  lengthy  to 
administer  and  SMEs  felt  its  lack  of  face  validity  was  problematic.  SYNWIN  garnered  positive 
reactions  and  requires  less  administration  time  than  the  ATSAT  subtests.  SMEs  noted  that 
SYNWIN  does  not  include  a  significant  psychomotor  component,  but  that  is  exactly  the  type  of 
measure  we  were  seeking.  We  contacted  Navy  researchers  to  find  out  if  the  SYNWIN  code 
would  be  available  to  the  Air  Force,  with  no  success.  Therefore,  we  recommended  programming 
an  Air  Force-owned  measure  that  would  bear  surface  similarities  to  SYNWIN,  but  would  also  be 
programmed  in  a  flexible  modular  format  that  the  Air  Force  could  change  the  type  of  tasks  that 
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be  performed  concurrently. 


Best  Bet  Predictor  Battery 

After  identifying  best-bet  predictors  for  each  SAOC,  including  possible  measurement 
gaps,  our  next  challenge  was  to  recommend  a  battery  of  predictors  that  could  potentially  be 
administered  in  an  entry-level  selection  process.  We  knew  that  we  had  to  work  within  practical 
constraints,  given  that  thousands  of  potential  candidates  would  need  to  complete  the  battery,  and 
potentially  at  many  different  locations.  The  first  practical  consideration  was  assembling  a 
predictor  battery  that  would  require  little  or  no  change  to  existing  entry-level  selection 
administration  procedures.  For  example,  it  would  not  be  realistic  to  suggest  a  battery  requiring 
specialized  equipment  or  hand  scoring  of  verbal  or  written  responses  or  the  presence  of  external 
observers.  We  also  learned  from  the  Air  Force  Recruiting  Service  that  it  would  not  be  realistic  to 
ask  recruiters  to  administer  and  score  assessments.  Therefore,  we  assumed  that  any  new 
measures  would  need  to  be  administered  on  existing  testing  platforms  using  existing  procedures 
(e.g.,  TBAS,  AFOQT,  and  ASVAB).  We  did,  however,  consider  the  possibility  of  making  a  P-E 
fit  measure  available  as  a  self-assessment  on  an  Air  Force  recruiting  website,  rather  than  as  an 
actual  screening  or  selection  tool. 

The  second  practical  consideration  was  assembling  a  predictor  battery  that  could  be 
administered  in  a  reasonable  amount  of  time.  For  example,  we  knew  it  would  be  impractical  to 
measure  all  of  the  critical  SAOCs  because  doing  so  would  require  several  hours  of  testing  time 
per  examinee.  We  gave  highest  priority  to  a  battery  of  assessments  that  would  measure 
fundamental  SAOCs  least  likely  to  be  impacted  by  post-enlistment/post-accessioning  training. 

The  third  practical  consideration  was  taking  advantage  of  the  voluminous  research  that 
the  Air  Force  and  the  other  services  have  already  devoted  to  developing  entry-level  selection 
measures.  As  shown  in  Table  5,  there  are  existing,  well-researched  measures  that  tap  many  of 
the  SAOCs  critical  for  RPA  pilots  and  SOs.  It  would  not  be  realistic  to  propose  replacing  these 
measures  with  others  that  might  add  little  to  overall  prediction  of  success  in  the  RPA  career 
fields.  Therefore,  we  gave  higher  priority  to  assessments  already  owned  by  the  USAF  or 
Department  of  Defense  than  to  commercially-available  assessments  or  assessments  that  would 
likely  be  costly  or  difficult  for  the  USAF  to  access. 

We  took  two  approaches  when  thinking  about  a  best  bet  predictor  battery.  The  first 
approach  focused  on  making  maximal  use  of  existing  predictor  measures  readily  available  to  the 
USAF.  This  battery  would  be  the  easiest  to  implement.  The  second  approach  focused  on 
identifying  a  battery  of  existing  measures  likely  to  demonstrate  smaller  gender  subgroup 
differences  than  existing  batteries  such  as  the  AFOQT  or  ASVAB.  The  second  approach  takes 
advantage  of  rigorous  research  on  non-verbal  measures  of  cognitive  ability  (see  Waters,  Russell, 
&  Sellman,  2007  for  a  review),  measures  of  spatial  and  psychomotor  abilities  (e.g.,  Ackennan  & 
Cianciolo,  2000;  Alderton  et  al.,  1997;  Russell  et  al.,  2001),  and  measures  of  perceptual  speed 
(Ackerman,  2004;  Ackerman  &  Beier,  2007).  Table  6  shows  our  final  recommendations. 
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Table  6.  Recommendations  for  an  Entry-level  RPA  Pilot  and  SO  Predictor  Battery 


Option  1 :  RPA  Pilot 

AFOQT 

—  Arithmetic  Reasoning 

—  Table  Reading 

—  Block  Counting 

—  Rotated  Blocks 

—  Hidden  Figures 

—  SD1+ 

TBAS 

New  measure  of  Time  Sharing  ability  that  can  be 
administered  on  the  TBAS  platform 

New  P-E  fit  measure  customized  for  RPA  environment 

Critical  SAOCs  covered  by  this  battery: 
Initiative 

Assertiveness 

Decisiveness 

Self-Control 

Stress  Tolerance 

Number  Facility 

Working  Memory 

Task  Prioritization 

Selective  Attention 

Perceptual  Speed 

Pattern  Recognition 

Spatial  Orientation 

Visualization 

Oral  Comprehension 

Control  Precision 

Option  1:  RPA  Sensor  Operator 

Critical  SAOCs  covered  by  this  battery: 
Initiative 

ASVAB 

Assertiveness 

—  Arithmetic  Reasoning 

Decisiveness 

—  Assembling  Objects 

Self-Control 

Number  Facility 

TAPAS 

Working  Memory 

Task  Prioritization 

TBAS  (if  possible,  replace  rudder  with  trackball) 

Selective  Attention 

Perceptual  Speed 

ECAT  Mental  Counters  or 

Pattern  Recognition 

Army  SIFT  or  PC-based  measure  of  perceptual  speed 

Spatial  Orientation 

Visualization 

New  measure  of  Time  Sharing  ability  that  can  be 

Oral  Comprehension 

administered  on  the  TBAS  platform 

Control  Precision 

New  P-E  fit  measure  customized  for  RPA  environment 

Option  2:  For  Both  RPA  Pilots  and  Sensor  Operators 

Critical  SAOCs  covered  by  this  battery: 
Initiative 

ECAT  Figural  Reasoning  or  Abstract  Reasoning  Test  (ART) 

Assertiveness 

Decisiveness 

ASVAB  Assembling  Objects  test 

Self-Control 

Number  Facility 

TAPAS 

Working  Memory 

Task  Prioritization 

TBAS  Tracking  subtests  (perhaps  replacing  rudder  pedals 

Selective  Attention 

with  trackball)  or  a  PC-based  psychomotor  test 

Perceptual  Speed 

Spatial  Orientation 

ECAT  Mental  Counters  or 

Pattern  Recognition 

Army  SIFT  or  PC-based  measure  of  perceptual  speed 

New  measure  of  Time  Sharing  ability  that  can  be 
administered  on  the  TBAS  platform 

New  P-E  fit  measure  customized  for  RPA  environment 

Visualization 

Control  Precision 
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Summary  of  Predictor  Battery  Recommendations 

Each  recommended  test  battery  covers  most  of  the  critical  SAOCs  identified  in  Table  3,  but 
not  one  covers  all  of  them.  The  SAOCs  that  were  dropped  from  further  consideration,  including  oral 
expression,  oral  comprehension,  critical  thinking  skills,  judgment  and  decision  making  skills,  and 
teamwork  skills  are  not  covered  in  any  of  the  batteries.  Each  battery  also  covers  all  of  the  major 
taxonomic  domains,  including  cognitive  abilities  and  skills,  work  style  (temperament)  characteristics, 
cognitive  processing  abilities  and  skills,  and  psychomotor  skills.  Option  1  makes  maximal  use  of 
existing  measures  for  both  RPA  pilots  and  SOs  while  Option  2  replaces  some  of  the  current 
operational  measures  with  measures  that  demonstrated  acceptable  levels  of  validity  and  smaller 
gender  subgroup  differences  than  some  of  the  existing  measures. 

For  RPA  Pilots,  the  core  of  the  Option  1  battery  includes  the  same  assessments  that  are 
already  used  in  manned  aircraft  pilot  training  candidate  selection.  It  includes  a  composite  of 
scores  on  the  AFOQT  subtests  and  TBAS.  The  AFOQT  subtests  recommended  for  selection  of 
RPA  pilots  do  not  mirror  the  existing  AFOQT  Pilot  composite.  At  this  point,  our 
recommendations  for  an  RPA  pilot  composite  of  AFOQT  scores  is  based  on  mapping  SAOC 
requirements  against  the  AFOQT  subtests.  Obviously,  research  would  be  required  to  determine 
the  appropriate  weighting  of  the  subtest  recommended  for  the  RPA  pilot  composite  and  to 
determine  if  it  provides  any  improvement  in  prediction  over  the  existing  AFOQT  Pilot 
composite.  A  new  measure  of  Time  Sharing  ability  could  be  incorporated  into  the  TBAS 
platform  with  relative  ease  because  no  new  hardware  or  peripheral  equipment  would  be  required. 
A  new  RPA-specific  P-E  fit  measure  could  be  administered  in  conjunction  with  the  TBAS  or  it 
as  a  self-assessment  available  to  officer  candidates  prior  to  joining  the  Air  Force. 

The  Option  1  battery  for  RPA  SOs  includes  two  measures  that  are  already  administered  at 
Military  Entrance  Processing  Stations  (MEPS)  and  Military  Enlistment  Test  Sites  (METS)-the 
ASVAB  and  the  TAPAS.  TBAS  is  currently  administered  to  enlisted  personnel  as  they  attend 
BMT  at  Randolph  AFB  and,  as  noted  above,  the  newly-developed  measure  of  Time  Sharing  ability 
could  be  administered  on  the  same  platform.  One  unknown  about  this  battery  is  whether  or  not  it  is 
feasible  to  administer  several  of  the  measures  post-enlistment,  and  then  use  the  results  to  support 
career  field  changes  for  enlisted  personnel  who  have  already  been  assigned  to  an  AFS  other  than 
1U0X1,  Unmanned  Aerospace  System  (UAS)  Sensor  Operator.  The  new  RPA-specific  P-E  fit 
measure  could  either  be  made  available  as  a  self-assessment  prior  to  enlistment  or  administered  on 
the  TBAS  platform. 

The  Option  2  battery  is  the  same  for  RPA  pilots  and  SOs.  It  replaces  existing  multi¬ 
aptitude  test  batteries  -  the  AFOQT  and  ASVAB  -  with  cognitive  ability  measures  that  focus 
more  heavily  on  fluid  intelligence  (nonverbal  reasoning  and  spatial  abilities).  These  measures 
have  shown  promise  for  predicting  important  outcomes  in  prior  studies  (e.g.,  training  or  job 
performance),  but  are  not  currently  used  for  operational  selection  by  the  U.S.  military.  Figural 
reasoning  tests  have  also  demonstrated  smaller  gender  subgroup  score  differences  and, 
sometimes,  smaller  race/ethnic  subgroup  score  differences  than  some  of  the  more  traditional 
cognitive  ability  tests  (Russell  et  ah,  2001;  Waters,  Russell,  &  Sellman,  2007).  The  ASVAB 

3  TAPAS  is  currently  under  trial  for  operational  use  in  U.S.  Army  enlisted  selection.  The  USAF  is  conducting  a  study  to 
evaluate  the  validity  of  TAPAS  for  prediction  of  performance  in  a  variety  of  AFSs. 
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Assembling  Objects  test  has  also  demonstrated  smaller  gender  differences  than  many  other 
measures  of  spatial  ability.  To  create  this  battery,  the  USAF  would  need  to  obtain  the  necessary 
permissions  and  determine  which  measures  should  be  administered  on  which  platform(s). 
TAPAS  is  already  administered  on  the  CAT-ASVAB  platform  at  the  MEPS  and  the  ART  may 
also  be  available  there  soon.  The  other  measures  could  potentially  be  incorporated  in  the  TBAS 
testing  platform. 

Development  and  Beta  Testing  of  a  New  Measure  of  Time  Sharing  Ability 

As  noted  above,  we  decided  to  develop  a  measure  of  Time  Sharing  ability  that  is  similar 
in  appearance  to  SYNWIN.  HumRRO  partnered  with  Aptima,  Inc.  to  develop  and  program  a 
multi-tasking  (MT)  test  that  provides  a  great  deal  of  flexibility  for  changing  test  parameters  and 
for  adding  different  tasks  in  the  future.  Currently,  it  includes  four  simple  tasks,  each  of  which  is 
easy  to  perform  by  itself.  Test-takers  are  provided  instruction  on  and  an  opportunity  to  practice 
performing  each  task  independently,  and  then  practice  performing  all  four  tasks  concurrently 
before  completing  the  operationally  scored  section  of  the  test.  The  number  and  length  of  the 
practice  trials  is  a  variable  parameter,  meaning  that  the  AFPC  can  change  the  parameters  prior 
to  any  particular  instance  of  testing.  The  MT  test  also  has  the  flexibility  to  change  virtually  any 
parameter  in  the  tasks  themselves,  such  as  the  rate  at  which  stimulus  objects  appear,  the  number 
of  digits  in  a  math  problem,  the  score  values  assigned  to  correct  and  incorrect  responses,  and  the 
way  in  which  a  non-response  is  treated.  Once  the  parameters  are  set,  it  is  expected  that  they  will 
remain  the  same  for  all  test-takers  until  an  authorized  test  administrator  changes  them.  We 
believe  this  tool  can  provide  a  foundation  for  extensive  future  research  on  Time  Sharing  ability, 
in  addition  to  providing  a  selection  instrument  for  RPA  pilot  and  SO  career  fields.  More  detail 
is  provided  about  this  measure  below,  as  well  as  in  the  Installation  and  Configuration  Guides 
that  were  delivered  to  the  AFPC  project  team. 

Overview  of  the  Multi-Tasking  (MT)  Test 

The  USAF  MT  Test  assesses  an  individual’s  skill  in  performing  multiple  tasks 
simultaneously.  Scores  on  the  MT  Test  are  intended  to  be  used  to  qualify  an  individual  for  entry- 
level  job(s)  requiring  multi-tasking  skills.  In  its  current  configuration,  the  MT  Test  (Figure  4) 
consists  of  the  following  four  tasks: 

•  Memorization  Task  (upper  left-hand  quadrant) 

•  Basic  Math  Skills  Task  (upper  right-hand  quadrant) 

•  Visual  Monitoring  Task  (lower  left-hand  quadrant) 

•  Listening  Task  (lower  right-hand  quadrant) 
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Figure  4.  Screenshot  of  the  MT  test. 

Memorization  Task 

What  Must  the  Examinee  Do? 

•  Memorize  a  series  of  letters  that  are  briefly  presented,  then  removed.  Then,  after 
a  brief  delay,  indicate  if  the  target  letter  shown  matches  one  of  the  series  in  the 
series,  now  hidden. 

•  Earn  points  for  correctly  answering  S  (“Yes”)  or  X  (“No”)  that  the  target  letter 
shown  matches  one  from  the  memorized  series. 

•  Lose  points  for  incorrectly  matching  the  target  letter  to  the  now  hidden  series  or 
for  failing  to  respond  before  time  expires.  Examinees  also  lose  points  every  time 
they  re-display  the  series. 

How  Does  the  Task  Work? 

•  A  series  of  letters  is  shown  in  the  top  box  (Figure  5).  The  test -taker  has  a  brief 
period  to  memorize  the  list  before  it  is  hidden. 
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Figure  5.  Memorization  task  item  -  list  shown. 


•  The  series  is  then  hidden  and  is  replaced  by  the  words  “Retrieve  List  (Lose 
Points)”  (Figure  6).  Clicking  “Retrieve  List  (Lose  Points)”  will  re-display  the 
series  of  letters.  However,  examinees  lose  points  every  time  they  click 
“Retrieve  List.” 


Retrieve  List  (Lose  Points) 

O 

Figure  6.  Memorization  task  item  -  list  hidden. 

•  After  another  brief  delay,  a  target  letter  appears  in  the  middle  box. 
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Figure  7.  Memorization  task  item  -  target  letter  instance. 

The  test-taker  then  has  a  specified  period  to  indicate  if  the  target  letter  shown  matches 
one  of  the  letters  in  the  series,  now  hidden.  The  test-taker  clicks  S  (“Y es”)  if  the  target  letter  did 
appear  in  the  series.  He  or  she  clicks  “No”  if  the  letter  did  not.  A  progress  bar  running  across  the 
top  box  shows  the  examinee  how  much  time  remains  to  answer. 

After  the  examinee  responds  or  time  expires,  there  is  brief  delay  before  a  new 
Memorization  item  is  presented.  During  this  time  the  test-taker  can  concentrate  on  one  or  more 
of  the  other  tasks. 

Basic  Math  Skills  Task 

What  Must  the  Examinee  Do? 

•  Solve  addition  or  subtraction  problems. 

•  Earn  points  for  correctly  solving  a  problem. 

•  Lose  points  for  answering  incorrectly  or  failing  to  solve  a  problem  before  time 
expires. 

How  Does  the  Task  Work? 

•  A  math  problem  (addition  or  subtraction)  is  presented  (Figure  8).  Immediately 
below  the  problem  is  the  answer  line  where  an  examinee’s  answer  is 
displayed. 
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Answer  Line 


510 

818 


0.  1 

2  3  4  5  |  6  7 

8  9 

Submit 

Figure  8.  Basic  math  skills  item  -  answer  line  highlighted. 


•  Below  the  answer  line  is  a  row  of  number  buttons  (0-9)  (Figure  9).  The 
examinees  use  these  buttons  to  enter  their  answers. 


Answer 

Number 

Buttons 


Figure  9.  Basic  math  skills  item  -  answer  number  buttons  highlighted. 


36 


•  The  examinees  click  on  the  buttons  that  matches  their  answers.  A  cursor  box 
highlights  which  number  in  the  answer  line  is  currently  active.  The  cursor  box 
starts  on  the  far  right  end  of  the  answer  line  and  automatically  moves  to  the  left  as 
the  examinees  enter  their  answers. 

•  To  change  an  answer,  the  examinees  simply  click  on  the  number  in  the  answer 
line  that  they  want  to  change.  The  cursor  box  will  then  highlight  the  selected 
number.  The  examinees  then  click  on  the  button  matching  the  new  number  they 
want  to  enter.  A  progress  bar  running  across  the  answer  line  shows  how 
much  time  remains  to  answer. 

•  The  examinees  click  “Submit”  when  they  have  finished  entering  their  answer. 


After  the  examinee  clicks  "Submit"  or  time  expires,  there  is  a  brief  delay  before  a  new 
Basic  Math  Skills  item  appears.  During  this  time  the  examinee  can  concentrate  on  one  or  more  of 
the  other  tasks. 

Visual  Monitoring  Task 

What  Must  the  Examinee  Do? 

•  Monitor  a  needle  as  it  moves  across  a  gauge  from  right  to  left. 

•  Earn  points  by  stopping  the  needle  before  it  reaches  the  end  of  the  gauge  (on  the 
left).  The  examinee  can  click  anywhere  in  the  gauge  to  stop  the  needle. 

•  The  number  of  points  the  examinee  could  earn  progressively  increases  across  the 
length  of  the  gauge.  The  closer  the  needle  is  to  the  end  of  the  gauge  when  the 
examinee  stops  it,  the  more  points  are  earned. 

•  Lose  points  for  failing  to  stop  the  needle  before  it  reaches  the  end  of  the  gauge. 

How  Does  the  Task  Work? 

•  A  needle  moves  across  the  gauge  from  right  to  left  (Figure  10).  The  rate  at  which 
the  needle  moves  stays  the  same  across  the  length  of  the  gauge.  The  needle  will 
not  speed  up  or  slow  down.  However,  the  rate  at  which  the  needle  moves  could 
vary  from  item  to  item. 
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Figure  10.  Visual  monitoring  item. 


•  The  examineesclick  on  the  gauge  before  the  needle  reaches  the  end.  They  can 
click  anywhere  in  the  gauge  to  stop  the  needle.  It  is  not  necessary  to  click  on  the 
needle  to  stop  it. 

After  the  examinees  stop  the  needle  or  time  expires,  there  will  be  a  brief  delay  before  a 
new  Visual  Monitoring  item  starts.  During  this  time  the  examinees  can  concentrate  on  one  or 
more  of  the  other  tasks. 

Listening  Task 

What  Must  the  Examinee  Do? 

•  Listen  for  high  tones. 

•  Earn  points  for  correctly  identifying  a  high  tone. 

•  Lose  points  for  responding  to  a  low  high  tone  or  failing  to  respond  to  a  high  tone. 
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Figure  11.  Listening  task  item. 


How  Does  the  Task  Work? 

•  A  tone  is  briefly  sounded. 

•  The  examinees  click  on  the  “High  Tone”  button  if  they  hear  a  high  tone  (Figure 
11).  The  examinees  do  nothing  if  they  hear  a  low  tone.  Response  speed  does 
not  affect  their  score,  as  long  as  they  respond  before  the  next  tone  begins. 

There  is  brief  delay  between  Listening  items.  During  this  time  the  examinees 
can  concentrate  on  one  or  more  of  the  other  tasks. 

Beta  Testing 

The  MT  test  was  beta  tested  on  the  same  platform  that  is  used  to  administer  the  TBAS. 
Air  Force  personnel  were  trained  to  administer  the  MT  test  and  beta  test  data  were  collected 
and  analyzed  to  ensure  that  the  test  software  functioned  as  intended  on  the  TBAS  platform. 
Full-scale  field  testing  to  collect  pilot  data  for  finalizing  an  operational  version  of  the  MTT  was 
beyond  the  scope  of  this  contract. 

Development  and  Field  Test  of  a  P-E  Fit  Measure 
Instrument  Development 

We  started  with  the  list  of  work  context  factors  shown  in  Table  3  and  turned  them  into 
more  generic  work  context  statements.  The  list  of  statements  underwent  several  iterations  of 
internal  review,  with  the  goal  of  writing  statements  that  were  neutral  in  tone  and  understandable 
to  persons  with  no  RPA  knowledge  or  experience  while  also  accurately  reflecting  the  RPA  work 
context.  We  also  wrote  some  “reverse”-scored  items  for  inclusion  in  the  instrument  by  trying  to 
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write  items  that  were  non-characteristic  of  the  RPA  work  context,  as  shown  in  Table  3.  Our  idea 
was  that  the  reverse-scored  items  could  serve  as  foils,  preventing  respondents  from  earning  a 
high  score  simply  by  endorsing  every  statement  as  desirable. 

We  examined  several  response  formats  including: 

•  Likert  response  scale  (e.g.,  “strongly  disagree”  to  “strongly  agree”).  This  type  of 
scale  is  easy  for  respondents  to  understand,  but  is  also  relatively  easy  to  fake.  It 
also  would  be  possible  for  respondents  to  give  the  same  response  to  most  items, 
which  could  lead  to  little  information  about  the  relative  strength  of  their 
preferences.  Faking  would  be  a  serious  concern  if  this  measure  was  administered 
under  high  stakes  conditions  such  as  operational  screening. 

•  Ranking  (e.g.,  rank  the  statements  in  order  of  preference).  This  response  format  is 
one  way  to  force  respondents  to  indicate  which  statements  are  relatively  more  and 
less  preferred,  and  it  can  potentially  reduce  faking  somewhat  by  preventing 
respondents  from  strongly  endorsing  every  feature  they  feel  they  should  endorse. 
Ranking  would  work  well  if  there  were  only  a  few  work  context  features  available 
for  comparison. 

•  Forced-choice  response  format  (e.g.,  “pick  the  statement  in  this  pair  that  is  most 
true  of  you”).  This  type  of  scale  is  harder  to  program,  administer,  and  score,  but  is 
one  way  to  make  it  harder  for  respondents  to  fake. 

We  discussed  response  fonnat  options  with  the  AFPC  team  and  reached  consensus  that  a 
self-report  P-E  fit  measure  would  likely  be  administered  as  a  self-assessment  tool,  rather  than  as 
an  operational  screening  tool,  because  the  instrument  would  likely  be  easily  faked.  We  felt  it 
would  be  best  to  administer  it  in  a  low  stakes  setting  (pre-enlistment/accession)  in  which  there 
would  be  little  reason  for  respondents  to  fake.  Ultimately,  we  used  a  Likert  rating  scale,  shown 
below,  supplemented  by  a  ranking  exercise. 


A 

B 

c 

D 

E 

This  is  something  1 
would  actively  try 
to  avoid  in  a  job. 

This  is  something  1 
might  try  to  avoid 
in  a  job. 

1  don't  care  if  this  is 
part  of  my  job  or 
not. 

This  is  something  1 
miaht  look  for  in  a 
job. 

This  is  something  1 
would  actively  seek 
out  as  part  of  a  job. 

Our  goal  in  using  the  Likert  scale  was  to  capture  responses  akin  to  behavioral  intentions 
because  intentions  are  closer  to  actual  behavior  than  are  opinions  (Fishbein  &  Ajzen,  1975).  Of 
course  the  longer  the  time  between  when  applicants  complete  this  measure  and  when  they 
express  a  preference  for  a  career  field  has  some  bearing  on  how  proximal  the  responses  are. 
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We  conducted  two  structured  reviews  of  the  draft  list  of  statements  with  RPA  SMEs: 


•  In  August  2010,  the  7 1 1 th  Human  Performance  Wing,  Human  Effectiveness 
Directorate,  Warfighter  Readiness  Research  Division,  Training  Systems  and 
Performance  Measurement  Branch  (711  HPW/RHAS)  in  Mesa,  AZ  hosted  focus 
groups  with  eight  experienced  ANG  operators,  and  one  contractor  who  was  a 
former  RPA  pilot  and  trainer.  We  asked  them  to  judge,  for  each  statement, 
whether  or  not  it  was  descriptive  of  the  RPA  work  context  (content  validity 
check),  and  whether  they  thought  most  applicants  would  view  the  statement  as  a 
desirable,  neutral,  or  undesirable  feature  of  the  RPA  environment.  We  also  asked 
them  to  tell  us  about  work  context  factors  that  were  missing  from  the  list.  We 
turned  a  number  of  their  suggestions  into  statements  that  were  incorporated  in  the 
draft  list  before  the  next  SME  review,  and  we  reworded  several  other  statements 
to  reflect  their  input. 

•  In  February  2011,  the  432d  Wing  at  Creech  AFB,  Nevada  scheduled  focus  groups 
with  21  Active  Duty  RPA  pilots  (10  early  career  and  1 1  senior)  and  15  RPA  SOs 
(8  early  career  and  7  senior).  We  met  separately  with  each  of  four  groups-  early 
career  pilots,  senior  pilots,  early  career  SOs,  and  senior  SOs-and  each  group 
evaluated  the  statements  in  the  draft  P-E  fit  measure.  We  asked  early  career  pilots 
and  SOs  to  evaluate  the  level  of  desirability  of  each  statement  because  we  felt 
they  would  be  closest  to  the  target  population  in  terms  of  their  knowledge  of  how 
persons  unfamiliar  with  the  RPA  environment  would  interpret  the  statements. 

Then  we  engaged  in  a  group  discussion,  soliciting  suggestions  to  make  the 
statements  more  clear  and  suggestions  for  additional  items.  We  asked  senior 
pilots  and  SOs  to  judge  whether  or  not  each  statement  was  descriptive  (or  not)  of 
the  RPA  work  environment,  then  engaged  in  a  discussion  to  clarify  the  wording  of 
the  statements  and  to  suggest  new  statements. 

HumRRO  incorporated  reviewer  comments  in  the  measure  and  then  asked  two  additional 
RPA  SMEs  to  review  it-one  was  the  same  contractor  who  had  participated  in  focus  groups  in 
August  2010;  the  other  was  a  senior  ANG  SO  referred  by  one  of  the  ANG  participants.  After  the 
SME  reviews,  we  abandoned  the  attempt  to  include  reverse-scored  or  non-characteristic  items. 
When  they  encountered  these  reverse  items,  SMEs  would  almost  invariably  say,  “Well,  that 
could  be  true  in  some  units  or  some  situations”  or  they  felt  the  statement  was  confusing. 
Therefore,  we  dropped  the  few  “reverse”  items  from  the  item  pool.  We  incorporated  all  of  the 
SME  comments  and  then  created  a  draft  instrument,  including  instructions,  which  was  reviewed 
by  the  AFPC  team  in  May  2011.  Their  comments  were  incorporated  in  the  field  test  version  of 
the  instrument  which  is  shown  in  Appendix  C.  It  was  called  the  Work  Interest  Inventory  (WII) 
during  the  field  test  exercise. 
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Field  Test  of  the  P-E  Fit  Measure 


Purpose  and  Sample 

The  purpose  of  the  field  test  was  to  collect  initial  data  and  reactions  on  the  measure  for 
use  in  recommending  potential  revisions  to  its  content  and  instructions  prior  to  a  larger-scale 
pilot  test. 

The  WII  (P-E  fit  measure)  was  administered  in  paper-and-pencil  fonnat  to  a  sample  of 
enlisted  trainees  (n  =  93;  52  males,  41  females;  mean  age  =  21.3  years)  completing  their  Basic 
Military  Training  (BMT)  at  Lackland  AFB,  TX  and  to  a  sample  of  officer  trainees  completing 
Officer  Training  School  (OTS)  at  Maxwell  AFB,  AL  (n  =  101;  85  males,  13  females;  mean 
age  =  27.8  years). 

The  measure  included  three  sections: 

•  Section  A.  Likert  response.  Respondents  rated  the  extent  to  which,  in  their  ideal 
job,  they  would  approach  or  avoid  each  statement,  using  the  Likert  scale  shown 
above. 

•  Section  B.  Ranking  exercise.  Respondents  selected,  in  rank  order,  the  five  most 
preferred  statements  for  an  ideal  job,  and  the  five  least  preferred. 

•  Section  C.  Reactions.  Respondents  rated  the  difficulty  of  doing  the  ranking 
exercise  and  their  general  reactions  to  the  instrument. 


Analysis  Approach 

There  were  two  steps  in  our  analyses.  First,  we  screened  for  and  removed  anomalous 
cases  or  data  points.  Second,  we  computed  basic  item  and  person  (scale)-level  statistics  for  use  in 
evaluating  the  initial  pool  of  items  and  the  feasibility  of  the  ranking  exercise. 

We  screened  for  excessive  missing  data  and  flat  responding  (i.e.,  marking  the  same 
response  option  for  all  characteristics).  We  also  screened  for  logical  inconsistencies  between  the 
“most  ideal”  and  “worst  item”  characteristics  for  Section  B  (i.e.,  choosing  a  characteristic  in 
both  sets  of  rankings),  as  well  as  between  Section  B  ranking  and  Section  A  ratings  (e.g., 
choosing  a  characteristic  as  among  one’s  top  five  “most  ideal”  in  Section  B,  but  rating  that 
characteristic  as  something  one  would  avoid  in  Section  A).  Flagged  cases  were  only  excluded 
from  analyses  specific  to  the  WII  section  in  which  the  anomaly  was  observed. 

Cases  were  filtered  from  analyses  on  the  grounds  of  missing  data  based  on  criteria 
appropriate  for  each  section  (i.e.,  missing  20%  or  more  Section  A  ratings,  failing  to  rank  five 
characteristics  for  both  the  “most  ideal”  and  “worst  item”  lists  in  Section  B,  missing  responses  to 
all  three  rating-scale  items  in  Section  C).  A  case  was  filtered  from  analyses  on  the  grounds  of  flat 
responding  for  Section  A  if  the  respondent  provided  the  same  response  to  all  37  items.  With 
regard  to  filtering  on  the  basis  of  anomalies  associated  with  response  inconsistencies,  logical 
inconsistencies  was  used  as  grounds  to  further  explore  the  data  for  a  given  case  for  evidence  that 
the  respondent’s  data  might  be  invalid  (e.g.,  the  respondent  did  not  understand  the  instructions  or 
was  not  motivated).  For  instance,  if  discrepancies  were  found  between  the  rankings  provided  in 
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Section  B  and  ratings  provided  in  Section  A  for  a  case,  we  closely  examined  the  item-level  data 
associated  with  that  case  for  any  further  anomalies  (e.g.,  seemingly  random  or  oddly-pattemed 
responding). 

Overall,  the  number  of  cases  excluded  from  analyses  was  limited  in  both  samples  and 
specific  to  selected  sections.  No  cases  from  either  sample  were  dropped  from  our  analyses 
completely. 

As  discussed  in  greater  detail  below,  we  examined  item-level  descriptive  statistics  for 
each  section  (e.g.,  mean,  standard  deviation,  minimum,  and  maximum).  Similarly,  we  computed 
scale-level  descriptive  statistics  across  respondents  for  each  section  (e.g.,  e.g.,  mean,  standard 
deviation,  minimum,  and  maximum).  Given  the  intended  use  of  the  WII  as  a  procedure  for 
gauging  applicants’  fit  to  the  RPA  work  context  and  providing  feedback  to  the  respondents  based 
on  their  responses,  we  examined  two  types  of  scale  scores  for  Section  A.  First,  we  computed  the 
average  response  over  the  37  items  for  each  respondent.  Because  all  items  are  scaled  to  be 
characteristic  of  the  RPA  job,  the  average  response  can  be  viewed  as  a  global  index  of  the  extent 
to  which  the  applicant  would  seek  out  job  attributes  that  are  relevant  to  the  RPA  job.  Second,  we 
computed  the  number  of  times  the  respondent  endorsed  each  of  the  five  response  options  over 
the  37  items  (the  number  of  “A”  responses,  the  number  of  “B”  response,  and  so  forth).  This 
information  can  be  used  to  provide  the  applicants  with  feedback  concerning  their  profile 
nonned  to  a  larger  sample  of  interest  (e.g.,  RPA  job  incumbents).  We  also  computed  the 
frequency  and  percentage  of  missing  or  invalid  responses  for  each  section. 

Field  Test  Results 

SECTION  A:  RATING  YOUR  IDEAL  JOB  -  LIKERT  RATING  EXERCISE 

Enlisted  Basic  (BMT)  Trainees.  Tables  7  and  8  summarize  comparisons  between  male 
and  female  enlisted  basic  trainees  in  their  ratings  of  the  37  characteristics.  At  the  item  level 
(Table  7),  only  six  out  of  the  37  items  evidenced  sizable  and  statistically  significant  gender 
differences  ( d  >  .50;  p  <  .05).  Specifically,  male  trainees  rated  items  4  (“responsibility  for 
reconnaissance  and  surveillance”),  13  (“taking  lethal  action  against  enemy  targets”),  and  14 
(“participating  in  combat  operations  from  a  remote  location”)  significantly  higher,  on  average, 
than  female  trainees,  while  females  rated  items  7  (“frequent  observation  and  monitoring  of  your 
actions  by  others”),  10  (“working  as  part  of  a  team”),  and  15  (“performing  many  tasks 
simultaneously”)  higher,  on  average,  than  male  trainees.  Similarly,  few  sizable  or  statistically 
significant  gender  differences  emerged  at  the  person-  or  scale-level,  as  shown  in  Table  8.  Both 
male  and  female  trainees,  for  example,  utilized  the  extreme  ends  of  the  response  scale  (i.e.,  88% 
and  70.7%  of  male  and  female  trainees,  respectively,  rated  at  least  one  characteristic  an  “A  - 
This  is  something  I  would  actively  try  to  avoid  in  a  job,”  and  94%  and  95.1%  of  males  and 
female  trainees,  respectively,  rated  at  least  one  characteristic  an  “E  -  This  is  something  I  would 
actively  seek  out  as  part  of  a  job”). 

Tables  9  and  10  summarize  the  corresponding  results  for  the  total  enlisted  trainee  sample. 
Overall,  the  item  means  were  in  the  expected  direction  and  demonstrated  sufficient  variability, 
with  standard  deviations  of  .80  or  greater  (see  Table  9).  Similarly,  none  of  the  items  exhibited 
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extremely  low  or  high  mean  ratings.  The  lowest  item  mean,  -1.09,  was  observed  for  item  32 
(“Potentially  limiting  the  amount  of  time  you  spend  with  family  or  friends”),  whereas  the  highest 
item  mean,  1.36,  was  observed  for  item  26  (“Opportunity  to  work  in  an  emerging  career  field”). 
Basic  trainees  tended  to  utilize  the  minimum  and  maximum  response  options  (80.2%  of  trainees 
rated  at  least  one  characteristic  as  “A  -  This  is  something  I  would  actively  try  to  avoid  in  a  job,” 
and  94.5%  of  trainees  rated  at  least  one  characteristic  as  “E  -  This  is  something  I  would  actively 
seek  out  as  part  of  a  job”).  Similar  properties  and  patterns  were  observed  at  the  scale-  level  (see 
Table  10). 

The  results  summarized  in  Table  10  can  be  viewed  as  a  profile  reflecting  how  typical  or 
average  basic  trainees  endorse  characteristics  that  are  descriptive  of  an  RPA  position  in  their  ideal 
job.  Figure  12  compares  this  average  response  profile  for  enlisted  basic  trainees  (grey)  to  the 
theoretically-expected  response  profile  of  a  hypothetical  applicant  whose  work  preferences  are 
incongruent  with  the  characteristics  of  the  RPA  job  (green).4  As  illustrated  in  Figure  12,  the 
basic  trainees  in  the  field  test  sample  seem  to  be  significantly  more  inclined  toward  an  RPA  job, 
on  average,  than  someone  whose  responses  would  be  incongruent  with  this  career  field.  This 
figure  illustrates  the  potential  value  of  the  WII  for  expanding  the  RPA  training  pipeline  by  better 
identifying  applicants  or  recruits  who  would  be  a  good  match  for  an  RPA  position. 

Officer  (OTS)  Trainees.  Tables  1 1  and  12  summarize  comparisons  between  male  and 
female  officer  trainees  in  their  ratings  of  the  37  characteristics.  At  the  item  level  (Table  1 1),  none 
of  the  items  exhibited  sizable  or  statically  significant  gender  differences  ( d  >  .50,  p  <  .05), 
although  the  latter  result  was  generally  because  of  low  power  (very  few  females  in  the  sample). 
Three  of  the  37  items  evidenced  a  standardized  mean  difference  (or  d)  greater  than  or  around  .50. 
Female  trainees  rated  items  1  (“making  decisions  that  significantly  impact  the  well-being  of 
others”)  and  30  (“few  opportunities  to  get  to  know  your  co-workers”)  more  positively  than 
male  trainees,  whereas  male  trainees  rated  item  34  (“sensing  and  reacting  to  the  urgency  and 
danger  faced  by  individuals  in  harm’s  way”)  higher,  on  average,  than  female  trainees.  Similarly, 
sizable  or  significant  gender  differences  at  the  person-  or  scale-level  were  minimal  (see  Table 
12).  Overall,  male  officer  trainees’  mean  rating  of  all  37  characteristics  (M=  .39,  SD  =  .31)  was 
equivalent  to  that  of  female  trainees  (M=  .38,  SD  =  .51)  (<7  =  .03,  ns). 

Tables  13  and  14  summarize  the  corresponding  item  and  scale-level  results  for  the  total 
officer  trainee  sample.  As  in  the  enlisted  trainee  sample,  item  means  were  generally  in  the 
expected  direction  and  the  items  showed  sufficient  variability,  with  most  item  standard 
deviations  being  .80  or  greater  (see  Table  13).  Only  a  few  items  had  item  means  greater  than 
+/-  1.00.  The  lowest  item  mean,  -1.20,  was  observed  for  item  32  (“potentially  limiting  the 
amount  of  time  you  spend  with  family  or  friends”),  while  the  highest  item  mean,  1.61,  was 
associated  with  item  17  (“opportunities  to  work  with  cutting-edge  aircraft  technology”).  Most 
of  the  items  exhibited  a  range  of  -2  to  +2.  None  of  the  items  had  a  range  uniformly  in  the 
positive  or  negative  end  of  the  rating  scale.  Comparable  properties  and  patterns  emerged  at  the 
scale-level,  as  reported  in  Table  14. 


4  The  hypothetical  applicant  profile  is  provided  for  illustration  purposes  only.  It  is  simply  a  profile  that  is  skewed 
toward  the  “avoid”  portion  of  the  scale  for  many  of  the  items  in  the  P-E  fit  measure.  It  is  not  based  on  actual  data 
from  applicants  or  SMEs. 
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Table  7.  Section  A:  Rating  Your  Ideal  Job  -  Item-Level  Comparison  between  Male  and 
Female  Enlisted  Basic  Trainees 


Item 

Male 

Female 

Male-Female 

Mean 

SD 

n 

Mean 

SD 

n 

Diff 

d 

t 

P 

1 

1.02 

.87 

47 

1.29 

.87 

41 

-.27 

-.31 

-1.46 

.15 

2 

1.27 

1.01 

48 

1.15 

.88 

41 

.12 

.13 

.62 

.54 

3 

-.77 

.95 

48 

-.51 

.95 

41 

-.26 

-.28 

-1.28 

.20 

4 

1.06 

1.10 

48 

.29 

.75 

41 

.77 

.82 

3.79 

.00 

5 

.44 

1.09 

48 

.49 

.90 

41 

-.05 

-.05 

-.24 

.81 

6 

.48 

.83 

48 

.22 

1.06 

41 

.26 

.28 

1.30 

.20 

7 

-.71 

.99 

48 

-.07 

.88 

41 

-.64 

-.68 

-3.18 

.00 

8 

.06 

1.04 

48 

.34 

.99 

41 

-.28 

-.28 

-1.29 

.20 

9 

.29 

.92 

48 

.17 

1.36 

41 

.12 

.11 

.50 

.62 

10 

.92 

.90 

48 

1.39 

.70 

41 

-.47 

-.59 

-2.74 

.01 

11 

.60 

1.05 

48 

.83 

.92 

41 

-.23 

-.23 

-1.07 

.29 

12 

.75 

1.02 

48 

1.02 

.94 

41 

-.27 

-.28 

-1.31 

.19 

13 

.94 

1.36 

48 

.42 

1.12 

41 

.52 

.42 

1.96 

.05 

14 

.92 

1.33 

48 

.34 

1.24 

41 

.58 

.45 

2.10 

.04 

15 

.52 

.85 

48 

.93 

.96 

41 

-.41 

-.46 

-2.12 

.04 

16 

.06 

.97 

47 

.46 

.95 

41 

-.40 

-.42 

-1.95 

.05 

17 

1.27 

.98 

48 

1.02 

1.13 

41 

.25 

.24 

1.10 

.27 

18 

.33 

.98 

48 

.15 

1.13 

41 

.19 

.18 

.84 

.40 

19 

.38 

1.06 

48 

.56 

1.16 

41 

-.19 

-.17 

-.79 

.43 

20 

.77 

.83 

48 

.76 

.99 

41 

.01 

.02 

.08 

.94 

21 

.50 

1.19 

48 

.54 

1.21 

41 

-.04 

-.03 

-.14 

.89 

22 

-.92 

1.38 

48 

-.51 

1.27 

41 

-.40 

-.31 

-1.43 

.16 

23 

.23 

.97 

48 

.42 

.89 

41 

-.19 

-.20 

-.93 

.35 

24 

.23 

1.31 

48 

-.29 

1.25 

41 

.52 

.41 

1.91 

.06 

25 

.90 

1.10 

48 

.93 

.93 

41 

-.03 

-.03 

-.14 

.89 

26 

1.27 

.82 

48 

1.45 

.78 

40 

-.18 

-.23 

-1.04 

.30 

27 

.63 

1.06 

48 

.85 

1.15 

40 

-.23 

-.21 

-.95 

.34 

28 

.23 

.91 

48 

.30 

1.02 

40 

-.07 

-.07 

-.35 

.73 

29 

.29 

.82 

48 

.23 

.86 

40 

.07 

.08 

.37 

.71 

30 

-.54 

.92 

48 

-.60 

1.01 

40 

.06 

.06 

.28 

.78 

31 

-.56 

.87 

48 

-.60 

.84 

40 

.04 

.04 

.20 

.84 

32 

-1.08 

.94 

48 

-1.10 

.81 

40 

.02 

.02 

.09 

.93 

33 

-.42 

1.11 

48 

-.68 

1.14 

40 

.26 

.23 

1.07 

.29 

34 

.69 

.88 

48 

.68 

1.19 

40 

.01 

.01 

.06 

.95 

35 

.88 

.73 

48 

1.05 

.78 

40 

-.18 

-.23 

-1.08 

.28 

36 

1.17 

.81 

48 

1.13 

.88 

40 

.04 

.05 

.23 

.82 

37 

.50 

.92 

48 

.48 

1.30 

40 

.03 

.02 

.11 

.92 

Note.  Diff=  raw  difference  in  item  means,  d  =  standardized  mean  difference  or  Cohen’s 
d.  Negative  D  iff  and  d  values  mean  that  female  enlisted  trainees  rated  an  item  higher,  on 
average,  than  male  trainees.  Statistically  significant  d  values  ( t  >  1-96,  p  <  .05)  are  in  bold 
typeface. 
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Table  8.  Section  A:  Rating  Your  Ideal  Job  -  Scale-Level  Comparison  between  Male  and 
Female  Enlisted  Basic  Trainees 


Statistic 

Male 

Female 

Male-Female 

Mean 

SD 

n 

Mean 

SD 

n 

Diff 

d 

t 

P 

Count:  Response  A 

2.67 

2.01 

48 

2.80 

2.52 

40 

-.13 

-.06 

-.28 

.78 

Count:  Response  B 

5.58 

3.86 

48 

4.90 

2.85 

40 

.68 

.20 

.93 

.36 

Count:  Response  C 

11.04 

6.05 

48 

11.65 

5.97 

40 

-.61 

-.10 

-.47 

.64 

Count:  Response  D 

9.85 

4.45 

48 

9.30 

3.75 

40 

.55 

.14 

.62 

.53 

Count:  Response  E 

7.81 

6.28 

48 

8.35 

6.80 

40 

-.54 

-.08 

-.39 

.70 

Percent:  Response  A 

.07 

.05 

48 

.08 

.07 

40 

.00 

-.06 

-.27 

.79 

Percent:  Response  B 

.15 

.10 

48 

.13 

.08 

40 

.02 

.20 

.94 

.35 

Percent:  Response  C 

.30 

.16 

48 

.31 

.16 

40 

-.02 

-.10 

-.47 

.64 

Percent:  Response  D 

.27 

.12 

48 

.25 

.10 

40 

.02 

.14 

.64 

.53 

Percent:  Response  E 

.21 

.17 

48 

.23 

.18 

40 

-.01 

-.08 

-.38 

.71 

Overall:  MEAN 

.39 

.43 

48 

.42 

.44 

41 

-.03 

-.06 

-.29 

.78 

Overall:  MDN 

.50 

.72 

48 

.66 

.73 

41 

-.16 

-.22 

-1.03 

.30 

Overall:  SD 

1.10 

.20 

48 

1.09 

.22 

41 

.01 

.05 

.24 

.81 

Overall:  MIN 

-1.90 

.31 

48 

-1.68 

.52 

41 

-.21 

-.51 

-2.38 

.02 

Overall:  MAX 

1.94 

.25 

48 

1.95 

.22 

41 

-.01 

-.06 

-.28 

.78 

Note.  Diff=  raw  difference  in  mean  scale  scores,  d  =  standardized  mean  difference  or  Cohen’s  d.  Negative  Diff  and 
d  values  mean  that  female  enlisted  trainees  endorsed  an  item  more  frequently,  on  average,  than  male  trainees. 
Statically  significant  d  values  ( t  >  1.96  ,p  <  .05)  are  in  bold  typeface. 
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Table  9.  Section  A:  Rating  Your  Ideal  Job  -  Item-Level  Statistics  for  Total  Enlisted  Basic  Trainee  Sample  (n 
=  91) 


Item 

Mean 

Median 

SD 

Min 

Max 

Skew 

Kurt 

Valid 

ResD 

Missing 

ResD 

% 

Valid 

1 

1.14 

1 

.88 

-2 

2 

-1.10 

1.28 

90 

1 

99 

2 

1.23 

1 

.94 

-2 

2 

-1.45 

2.15 

91 

0 

100 

3 

-.65 

-1 

.95 

-2 

1 

.04 

-.94 

91 

0 

100 

4 

.71 

1 

1.03 

-2 

2 

-.28 

-.77 

91 

0 

100 

5 

.47 

0 

.99 

-2 

2 

-.03 

-.44 

91 

0 

100 

6 

.36 

0 

.94 

-2 

2 

-.29 

.01 

91 

0 

100 

7 

-.44 

0 

.99 

-2 

2 

.42 

.05 

91 

0 

100 

8 

.19 

0 

1.01 

-2 

2 

.27 

.01 

91 

0 

100 

9 

.22 

0 

1.13 

-2 

2 

-.12 

-.57 

91 

0 

100 

10 

1.14 

1 

.84 

-1 

2 

-.51 

-.78 

91 

0 

100 

11 

.73 

1 

.99 

-2 

2 

-.54 

-.38 

91 

0 

100 

12 

.87 

1 

1.00 

-2 

2 

-.74 

-.14 

91 

0 

100 

13 

.71 

1 

1.27 

-2 

2 

-.61 

-.71 

91 

0 

100 

14 

.65 

1 

1.32 

-2 

2 

-.57 

-.81 

91 

0 

100 

15 

.71 

1 

.92 

-2 

2 

-.26 

-.33 

91 

0 

100 

16 

.22 

0 

.99 

-2 

2 

.03 

-.08 

90 

1 

99 

17 

1.18 

2 

1.05 

-2 

2 

-1.25 

1.05 

91 

0 

100 

18 

.26 

0 

1.05 

-2 

2 

-.38 

-.35 

91 

0 

100 

19 

.46 

0 

1.10 

-2 

2 

-.23 

-.37 

91 

0 

100 

20 

.76 

1 

.90 

-2 

2 

-.16 

-.36 

91 

0 

100 

21 

.53 

0 

1.19 

-2 

2 

-.35 

-.62 

91 

0 

100 

22 

-.75 

-1 

1.33 

-2 

2 

.74 

-.65 

91 

0 

100 

23 

.31 

0 

.93 

-2 

2 

-.14 

.07 

91 

0 

100 

24 

.03 

0 

1.32 

-2 

2 

.00 

-.99 

91 

0 

100 

25 

.92 

1 

1.01 

-1 

2 

-.56 

-.79 

91 

0 

100 

26 

1.36 

2 

.80 

-1 

2 

-.87 

-.42 

90 

1 

99 

27 

.76 

1 

1.10 

-2 

2 

-.47 

-.71 

90 

1 

99 

28 

.28 

0 

.96 

-2 

2 

-.12 

-.21 

90 

1 

99 

29 

.26 

0 

.84 

-2 

2 

.41 

.30 

90 

1 

99 

30 

-.54 

-1 

.98 

-2 

2 

.63 

.35 

90 

1 

99 

31 

-.58 

-1 

.85 

-2 

1 

-.03 

-.60 

90 

1 

99 

32 

-1.09 

-1 

.88 

-2 

2 

.78 

.45 

90 

1 

99 

33 

-.53 

-1 

1.11 

-2 

2 

.39 

-.36 

90 

1 

99 

34 

.69 

1 

1.02 

-2 

2 

-.62 

.03 

90 

1 

99 

35 

.96 

1 

.76 

-1 

2 

-.08 

-.88 

90 

1 

99 

36 

1.14 

1 

.84 

-1 

2 

-.63 

-.43 

90 

1 

99 

37 

.52 

1 

1.11 

-2 

2 

-.41 

-.38 

90 

1 

99 

Note.  Valid  Resp  =  Number  of  valid  responses,  Missing  Resp  =  Number  of  missing  responses,  % 
Valid  =  Percentage  of  valid  responses.  Items  scored  on  a  scale  from  -2  (“A:  This  is  something  I 
would  actively  try  to  avoid  in  a  job.”)  to  +2  (“E:  This  is  something  I  would  actively  seek  out  as 
part  of  a  job.”). 
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Table  10.  Section  A:  Rating  Your  Ideal  Job  -  Scale-Level  Descriptive  Statistics  for  Total  Enlisted  Basic 
Trainee  Sample  (n  =  90) 


Mean 

Median 

SD 

Min 

Max 

Skew 

Kurt 

Valid 

Resv 

Missing 

Resv 

% 

Valid 

Count:  Response  A 

2.71 

2.00 

2.24 

0 

9 

.68 

-.17 

90 

0 

100 

Count:  Response  B 

5.27 

5.00 

3.40 

0 

17 

.77 

.65 

90 

0 

100 

Count:  Response  C 

11.31 

11.00 

5.93 

0 

28 

.61 

.60 

90 

0 

100 

Count:  Response  D 

9.51 

10.00 

4.15 

0 

23 

.31 

.65 

90 

0 

100 

Count:  Response  E 

8.18 

6.50 

6.48 

0 

32 

1.09 

1.12 

90 

0 

100 

Percent:  Response  A 

7.3% 

5.4% 

6.1% 

0.0% 

24.3% 

.68 

-.18 

90 

0 

100 

Percent:  Response  B 

14.2% 

13.5% 

9.2% 

0.0% 

45.9% 

.77 

.65 

90 

0 

100 

Percent:  Response  C 

30.6% 

29.7% 

16.0% 

0.0% 

75.7% 

.60 

.60 

90 

0 

100 

Percent:  Response  D 

25.7% 

27.0% 

11.2% 

0.0% 

62.2% 

.30 

.66 

90 

0 

100 

Percent:  Response  E 

22.1% 

17.6% 

17.5% 

0.0% 

86.5% 

1.09 

l.n 

90 

0 

100 

Overall:  MEAN 

.41 

.38 

.43 

-.59 

1.49 

.09 

-.39 

90 

0 

100 

Note.  Valid  Resp  =  Number  of  valid  responses,  Missing  Resp  =  Number  of  missing  responses,  %  Valid  = 
Percentage  of  valid  responses.  Overall:  MEAN  =  Mean  rating  across  37  Section  A  items. 


16 

14 


/ 


/ 


Count: 
Response  A 


Count: 
Response  B 


Count: 
Response  C 


Count: 
Response  D 


Count: 
Response  E 


Figure  12.  A  comparison  between  the  average  response  profile  for  the  enlisted  basic  trainee  sample  (grey)  on 
Section  A  versus  the  response  profile  for  a  hypothetical  applicant  with  preferences  that  are  incongruent  with 
an  RPA  position  (green). 
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Table  11.  Section  A:  Rating  Your  Ideal  Job  -  Item-Level  Comparison  between  Male  and  Female  Officer 
Trainees  (OTS) 


Item 

Male 

Female 

Male-Female 

Mean 

SD 

n 

Mean 

SD 

n 

Diff 

d 

t 

P 

1 

1.14 

.79 

85 

1.54 

.78 

13 

-.40 

-.51 

-1.69 

.09 

2 

1.15 

.89 

85 

1.31 

.95 

13 

-.15 

-.17 

-.58 

.57 

3 

-.28 

.85 

85 

-.31 

.85 

13 

.03 

.03 

.10 

.92 

4 

.74 

.90 

85 

.69 

.75 

13 

.05 

.06 

.19 

.85 

5 

.02 

.89 

85 

.08 

.95 

13 

-.05 

-.06 

-.20 

.84 

6 

.21 

1.09 

85 

.15 

1.21 

13 

.06 

.05 

.18 

.86 

7 

-.66 

.98 

85 

-.46 

1.13 

13 

-.20 

-.20 

-.66 

.51 

8 

.09 

.93 

85 

.31 

1.11 

13 

-.21 

-.23 

-.75 

.46 

9 

-.13 

.92 

85 

.08 

1.44 

13 

-.21 

-.21 

-.69 

.49 

10 

1.27 

.81 

85 

1.31 

.95 

13 

-.04 

-.05 

-.15 

.88 

11 

.64 

.99 

85 

.46 

1.20 

13 

.17 

.17 

.57 

.57 

12 

1.22 

.70 

85 

1.08 

.95 

13 

.15 

.20 

.67 

.50 

13 

.71 

.97 

85 

.62 

1.26 

13 

.09 

.09 

.30 

.77 

14 

.51 

1.02 

85 

.69 

1.11 

13 

-.19 

-.18 

-.61 

.55 

15 

.75 

.83 

85 

.62 

1.39 

13 

.14 

.15 

.50 

.62 

16 

.25 

.84 

85 

.23 

1.09 

13 

.02 

.02 

.06 

.95 

17 

1.65 

.70 

85 

1.38 

.87 

13 

.26 

.37 

1.22 

.23 

18 

.29 

1.01 

85 

.23 

.93 

13 

.06 

.06 

.21 

.83 

19 

-.11 

1.07 

85 

-.15 

1.21 

13 

.05 

.04 

.15 

.88 

20 

.87 

.77 

85 

1.08 

.76 

13 

-.21 

-.27 

-.90 

.37 

21 

.40 

1.28 

84 

.08 

1.38 

13 

.33 

.26 

.85 

.40 

22 

-.94 

1.12 

85 

-1.00 

1.29 

13 

.06 

.05 

.17 

.86 

23 

.52 

.87 

85 

.62 

.87 

13 

-.10 

-.11 

-.38 

.71 

24 

.24 

1.05 

85 

-.23 

1.09 

13 

.47 

.44 

1.48 

.14 

25 

1.00 

.79 

85 

.77 

.93 

13 

.23 

.29 

.96 

.34 

26 

1.46 

.65 

85 

1.46 

.66 

13 

.00 

.00 

-.01 

.99 

27 

1.01 

.88 

85 

.77 

1.30 

13 

.24 

.26 

.86 

.39 

28 

.31 

.86 

85 

.15 

1.07 

13 

.15 

.17 

.57 

.57 

29 

-.15 

.75 

85 

-.15 

1.07 

13 

.00 

.00 

.00 

1.00 

30 

-1.07 

.77 

85 

-.62 

1.39 

13 

-.46 

-.53 

-1.76 

.08 

31 

-.58 

.88 

85 

-.31 

1.32 

13 

-.27 

-.29 

-.96 

.34 

32 

-1.21 

.80 

85 

-1.08 

1.26 

13 

-.13 

-.16 

-.52 

.61 

33 

-.32 

.97 

85 

-.31 

1.11 

13 

-.01 

-.01 

-.03 

.97 

34 

.61 

.73 

85 

.23 

1.09 

13 

.38 

.49 

1.64 

.10 

35 

.86 

.83 

85 

.69 

.85 

13 

.17 

.20 

.67 

.51 

36 

1.39 

.73 

85 

1.62 

.51 

13 

-.23 

-.33 

-1.09 

.28 

37 

.68 

.95 

85 

.62 

.87 

13 

.07 

.07 

.24 

.81 

Note.  Diff=  Raw  difference  in  item  means,  d  =  standardized  mean  difference  or  Cohen’s  d. 
Negative  Diff  and  d  values  mean  that  female  officer  trainees  rated  an  item  higher,  on  average, 
than  male  trainees.  Statistically  significant  d  values  ( t  >  1.96 ,p  <  .05)  are  in  bold  typeface. 
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Table  12.  Section  A:  Rating  Your  Ideal  Job  -  Scale-Level  Comparison  between  Male  and  Female  Officer 
Trainees  (OTS) 


Statistic 

Male 

Female 

Male-Female 

Mean 

SD 

n 

Mean 

SD 

n 

Diff 

d 

t 

P 

Count:  Response  A 

2.38 

1.96 

85 

2.92 

2.47 

13 

-.55 

-21 

-.90 

.37 

Count:  Response  B 

5.73 

2.87 

85 

6.38 

4.23 

13 

-.66 

-.22 

-.72 

.48 

Count:  Response  C 

10.67 

4.45 

85 

10.00 

4.16 

13 

.67 

.15 

.51 

.61 

Count:  Response  D 

11.40 

4.45 

85 

8.92 

3.23 

13 

2.48 

.58 

1.93 

.06 

Count:  Response  E 

6.81 

4.15 

85 

8.77 

6.71 

13 

-1.96 

-.43 

-1.45 

.15 

Percent:  Response  A 

.06 

.05 

85 

.08 

.07 

13 

-.01 

-.27 

-.90 

.37 

Percent:  Response  B 

.15 

.08 

85 

.17 

.11 

13 

-.02 

-.22 

-.72 

.48 

Percent:  Response  C 

.29 

.12 

85 

.27 

.11 

13 

.02 

.15 

.51 

.61 

Percent:  Response  D 

.31 

.12 

85 

.24 

.09 

13 

.07 

.58 

1.93 

.06 

Percent:  Response  E 

.18 

.11 

85 

.24 

.18 

13 

-.05 

-.43 

-1.44 

.15 

Overall:  MEAN 

.39 

.31 

85 

.38 

.51 

13 

.01 

.03 

.08 

.93 

Overall:  MDN 

.45 

.52 

85 

.62 

.77 

13 

-.17 

-.30 

-1.01 

.32 

Overall:  SD 

1.10 

.17 

85 

1.13 

.23 

13 

-.03 

-.17 

-.56 

.58 

Overall:  MIN 

-1.82 

.38 

85 

-1.77 

.60 

13 

-.05 

-.13 

-.44 

.66 

Overall:  MAX 

1.95 

.21 

85 

1.92 

.28 

13 

.03 

.14 

.45 

.65 

Note.  Diff=  Raw  difference  in  mean  scale  scores,  d  =  standardized  mean  difference  or  Cohen’s  d.  Negative  D  iff  and  d  values  mean 
that  female  officer  trainees  endorsed  an  item  more  frequently,  on  average,  than  male  trainees.  Statically  significant  d  values  (t  >  1.96, 
p  <  .05)  are  in  bold  typeface. 


Figure  12  compares  this  average  response  profile  for  enlisted  basic  trainees  (grey)  to  the  theoretically- 
expected  response  profile  of  a  hypothetical  applicant  whose  work  preferences  are  incongruent  with  the 
characteristics  of  RPA  job  (green). ?  As  illustrated  in  Figure  12,  the  basic  trainees  in  the  field  test  sample  seem 
to  be  significantly  more  inclined  toward  an  RPA  job,  on  average,  than  someone  whose  responses  would  be 
incongruent  with  this  career  field.  This  figure  illustrates  the  potential  value  of  the  WII  for  expanding  the  RPA 
training  pipeline  by  better  identifying  applicants  or  recruits  who  would  be  a  good  match  for  an  RPA  position. 


5  The  hypothetical  applicant  profile  is  provided  for  illustration  purposes  only.  It  is  simply  a  profile  that  is  skewed  toward  the  “avoid” 
portion  of  the  scale  for  many  of  the  items  in  the  P-E  fit  measure.  It  is  not  based  on  actual  data  from  applicants  or  SMEs. 
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Table  13.  Section  A:  Rating  Your  Ideal  Job  -  Item-Level  Statistics  for  Total  Officer  Trainee  Sample  (n  = 
101) 


Item 

Mean 

Median 

SD 

Min 

Max 

Skew 

Kurt 

Valid 

ResD 

Missing 

ResD 

% 

Valid 

1 

1.20 

1 

.79 

-1 

2 

-.62 

-.37 

101 

0 

100 

2 

1.16 

1 

.91 

-2 

2 

-1.12 

1.03 

101 

0 

100 

3 

-.28 

0 

.84 

-2 

2 

.36 

.43 

101 

0 

100 

4 

.71 

1 

.91 

-1 

2 

-.29 

-.66 

101 

0 

100 

5 

.03 

0 

.92 

-2 

2 

.02 

.09 

101 

0 

100 

6 

.20 

0 

1.10 

-2 

2 

-.17 

-.76 

101 

0 

100 

7 

-.64 

-1 

1.00 

-2 

2 

.16 

-.75 

101 

0 

100 

8 

.07 

0 

.99 

-2 

2 

.30 

.11 

101 

0 

100 

9 

-.14 

0 

1.01 

-2 

2 

.17 

-.59 

101 

0 

100 

10 

1.30 

1 

.82 

-1 

2 

-.94 

.11 

101 

0 

100 

11 

.61 

1 

1.00 

-2 

2 

-.32 

-.68 

101 

0 

100 

12 

1.21 

1 

.73 

-1 

2 

-.66 

.27 

101 

0 

100 

13 

.67 

1 

1.01 

-2 

2 

-.49 

-.10 

101 

0 

100 

14 

.51 

1 

1.05 

-2 

2 

-.31 

-.51 

101 

0 

100 

15 

.73 

1 

.90 

-2 

2 

-.35 

-.20 

101 

0 

100 

16 

.24 

0 

.87 

-2 

2 

-.12 

-.19 

101 

0 

100 

17 

1.61 

2 

.72 

-1 

2 

-1.88 

2.88 

101 

0 

100 

18 

.27 

0 

1.00 

-2 

2 

.05 

-.68 

101 

0 

100 

19 

-.16 

0 

1.10 

-2 

2 

.09 

-.64 

101 

0 

100 

20 

.89 

1 

.76 

-1 

2 

-.23 

-.34 

101 

0 

100 

21 

.36 

0 

1.31 

-2 

2 

-.37 

-.81 

100 

1 

99 

22 

-.94 

-1 

1.14 

-2 

2 

.79 

-.34 

101 

0 

100 

23 

.51 

1 

.89 

-2 

2 

-.52 

.14 

101 

0 

100 

24 

.17 

0 

1.06 

-2 

2 

-.04 

-.64 

101 

0 

100 

25 

.99 

1 

.81 

-1 

2 

-.45 

-.28 

101 

0 

100 

26 

1.45 

2 

.66 

0 

2 

-.77 

-.45 

101 

0 

100 

27 

.97 

1 

.94 

-2 

2 

-1.04 

1.02 

101 

0 

100 

28 

.29 

0 

.88 

-2 

2 

-.23 

.23 

101 

0 

100 

29 

-.15 

0 

.78 

-2 

2 

-.12 

.12 

101 

0 

100 

30 

-1.02 

-1 

.87 

-2 

2 

1.06 

1.72 

101 

0 

100 

31 

-.54 

-1 

.94 

-2 

2 

.42 

.17 

101 

0 

100 

32 

-1.20 

-1 

.86 

-2 

2 

.97 

.87 

101 

0 

100 

33 

-.30 

0 

.98 

-2 

2 

-.03 

-.25 

101 

0 

100 

34 

.55 

1 

.78 

-1 

2 

-.06 

-.35 

101 

0 

100 

35 

.82 

1 

.84 

-1 

2 

-.27 

-.52 

101 

0 

100 

36 

1.43 

2 

.70 

-2 

2 

-1.53 

4.50 

101 

0 

100 

37 

.66 

1 

.94 

-2 

2 

-.60 

.12 

101 

0 

100 

Note.  Valid  Resp  =  Number  of  valid  responses,  Missing  Resp  =  Number  of  missing  responses,  %  Valid  =  Percentage  of  valid 
responses.  Items  rated  on  a  scale  from  -2  (“A:  This  is  something  1  would  actively  try  to  avoid  in  a  job.”)  to  +2  (“E:  This  is  something  1 
would  actively  seek  out  as  part  of  a  job.”). 
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Table  14.  Section  A:  Rating  Your  Ideal  Job  -  Scale-Level  Descriptive  Statistics  for  Total 
Officer  Trainee  (OTS)  Sample  (n  =  101) 


Mean 

Median 

SD 

Min 

Max 

Skew 

Kurt 

Valid 

Resv 

Missing 

ResD 

% 

Valid 

Count:  Response  A 

2.51 

2.00 

2.19 

0 

11 

1.27 

2.12 

101 

0 

100 

Count:  Response  B 

5.85 

5.00 

3.03 

0 

16 

.77 

.84 

101 

0 

100 

Count:  Response  C 

10.52 

10.00 

4.37 

1 

22 

.05 

-.54 

101 

0 

100 

Count:  Response  D 

11.07 

11.00 

4.45 

2 

22 

.26 

-.32 

101 

0 

100 

Count:  Response  E 

7.03 

6.00 

4.54 

0 

22 

.80 

.66 

101 

0 

100 

Percent:  Response  A 

6.8% 

5.4% 

5.9% 

0.0% 

29.7% 

1.27 

2.11 

101 

0 

100 

Percent:  Response  B 

15.8% 

13.5% 

8.2% 

0.0% 

43.2% 

.77 

.84 

101 

0 

100 

Percent:  Response  C 

28.5% 

27.0% 

11.8% 

2.7% 

59.5% 

.06 

-.54 

101 

0 

100 

Percent:  Response  D 

29.9% 

29.7% 

12.0% 

5.4% 

59.5% 

.26 

-.32 

101 

0 

100 

Percent:  Response  E 

19.0% 

16.2% 

12.3% 

0.0% 

59.5% 

.80 

.65 

101 

0 

100 

Overall:  MEAN 

.39 

.41 

.35 

-.41 

1.46 

.28 

.74 

101 

0 

100 

Note.  Valid  Resp  =  Number  of  valid  responses,  Missing  Resp  =  Number  of  missing  responses,  %  Valid  = 
Percentage  of  valid  responses.  Overall:  MEAN  =  Mean  rating  across  37  Section  A  items. 


Figure  13.  A  comparison  between  the  average  response  profile  for  the  OTS  sample  (grey)  on 
Section  A  versus  the  response  profile  for  a  hypothetical  applicant  with  preferences  that  are 
incongruent  with  an  RPA  position  (green). 
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SECTION  B:  DESCRIBING  YOUR  IDEAL  AND  WORST  JOBS  -  RANKING 

EXERCISE 


Enlisted  Basic  (BMT)  Trainees.  Table  15  summarizes  the  characteristics  most  frequently 
ranked  by  enlisted  basic  trainees  in  their  top  five  for  describing  their  “Most  Ideal  Job”  and  their 
“Worst  Job”  by  gender.  As  reported  in  Table  15,  three  of  the  top  five  most  frequently  cited 
characteristics  for  describing  one’s  “Most  Ideal  Job”  were  the  same  for  males  and  females:  (a) 
opportunities  to  protect  or  rescue  friendly  forces  from  harm  (item  2);  (b)  opportunities  to  work 
with  cutting-edge  aircraft  technology”  (item  17);  and  (c)  opportunity  to  work  in  an  emerging 
career  field  (item  26).  Consistent  with  the  ratings  from  Section  A,  however,  male  trainees  were 
more  likely  to  rank  combat-related  characteristics  in  their  top  five  (e.g.,  taking  lethal  action 
against  enemy  targets,  item  13),  whereas  females  were  more  likely  to  cite  social  aspects  in  their 
top  five  (e.g.,  working  as  part  of  a  team,  item  10).  Results  were  similar  for  the  top  five 
characteristics  describing  one’s  “Worst  Job”,  with  four  of  the  top  five  most  frequently 
ranked  characteristics  being  the  same  for  males  and  females:  (a)  providing  negative  performance 
feedback  to  co-workers  (item  3);  (b)  spending  most  of  the  work  shift  in  a  windowless  room, 
viewing  multiple  computer  monitors  (item  22);  (c)  few  opportunities  to  get  to  know  your  co¬ 
workers  (item  30);  and  (d)  potentially  limiting  the  amount  of  time  you  spend  with  family  or 
friends  (item  32).  Male  trainees  more  frequently  ranked  “frequent  observation  and  monitoring  of 
your  actions  by  others”  (item  7)  among  the  top  five  for  describing  their  “Worst  Job”,  while 
female  trainees  cited  “performing  actions  that  could  seriously  injure  or  kill  others  (non-friendly 
and  friendly)”  (item  24). 

Officer  (OTS)  Trainees.  Table  16  summarizes  the  characteristics  most  frequently  ranked 
by  officer  trainees  in  their  top  five  for  describing  their  “Most  Ideal  Job”  and  their  “Worst  Job”  by 
gender.  Similar  to  the  enlisted  trainee  sample,  male  and  female  officer  trainees  ranked  the  same 
three  characteristics  among  their  top  five  for  describing  their  “Most  Ideal  Job”:  (a)  opportunities 
to  protect  or  rescue  friendly  forces  from  harm  (item  10);  (b)  opportunity  to  work  in  an  emerging 
career  field  (item  26);  and  (c)  ability  to  see  how  your  work  contributes  to  the  broader  mission 
(item  36).  The  two  groups  differed  in  that  females  were  more  likely  to  endorse  characteristics 
related  to  helping  others  (e.g.,  opportunities  to  protect  or  rescue  friendly  forces  from  harm,  item 
2).  Male  and  female  officer  trainees  similarly  ranked  the  same  four  characteristics  among  their 
top  five  for  describing  their  “Worst  Job”:  (a)  frequent  observation 

and  monitoring  of  your  actions  by  others  (item  7);  (b)  spending  most  of  the  work  shift  in  a 
windowless  room,  viewing  multiple  computer  monitors  (item  22);  (c)  few  opportunities  to  get  to 
know  your  co-workers  (item  30);  and  (d)  potentially  limiting  the  amount  of  time  you  spend  with 
family  or  friends  (item  32).  The  two  groups  differed  in  that  male  trainees  included  “limited 
interactions  with  co-workers  outside  of  work”  (item  31)  among  their  top  five,  while  female 
trainees  were  more  likely  to  rank  “remaining  alert  and  highly  focused  for  long  periods  of  time” 
(item  9)  in  their  top  five  for  describing  their  “Worst  Job.” 

Comparing  Enlisted  and  Officer  Trainees.  Table  17  compares  the  enlisted  basic  and 
officer  trainees  on  the  characteristics  most  frequently  ranked  in  their  top  five  for  describing  their 
“Most  Ideal  Job”  and  their  “Worst  Job”.  Overall,  enlisted  basic  and  officer  trainees  were 
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generally  consistent  in  the  characteristics  most  frequently  ranked  in  their  respective  top  five 
choices.  When  describing  their  “Most  Ideal  Job,”  enlisted  basic  and  officer  trainees  had  four 
characteristics  in  common  among  their  top  five:  (a)  opportunities  to  protect  friendly  forces  (item 
2);  (b)  working  as  part  of  a  team  (item  10);  (c)  opportunities  to  work  with  cutting-edge  aircraft 
technology  (item  17);  and  (d)  opportunity  to  work  in  an  emerging  career  field  (item  26).  The  two 
groups  differed  in  that  enlisted  basic  trainees  cited,  “taking  lethal  action  against  enemy  targets” 
(item  13),  among  their  top  five  for  describing  their  “Most  Ideal  Job,”  whereas  officer  trainees 
included,  “ability  to  see  how  your  work  contributes  to  the  broader  mission”  (item  36)  in  their  top 
five.  Enlisted  basic  and  officer  trainees  similarly  had  four  characteristics  in  common  among  the 
top  five  for  describing  their  “Worst  Job”:  (a)  frequent  observation  and  monitoring  of  your  actions 
by  others  (item  7);  (b)  spending  most  of  the  work  shift  in  a  windowless  room,  viewing  multiple 
computer  monitors  (item  22);  (c)  few  opportunities  to  get  to  know  your  co-workers  (item  30); 
and  (d)  potentially  limiting  the  amount  of  time  you  spend  with  family  or  friends  (item  32).  The 
two  groups  differed  in  that  enlisted  basic  trainees  ranked,  “providing  negative  performance 
feedback  to  co-workers”  (item  3),  among  their  top  five,  while  officer  trainees  cited,  “remaining 
alert  and  highly  focused  for  long  periods  of  time”  (item  9)  for  describing  their  “Worst  Job.”  Both 
enlisted  basic  and  officer  trainees  ranked  “potentially  limiting  the  amount  of  time  with  family  or 
friends”  (item  32)  and  “spending  most  of  the  work  shift  in  a  windowless  room,  viewing  multiple 
computer  monitors”  (item  22)  as  the  two  characteristics  most  descriptive  of  their  “Worst  Job.” 
Over  50%  of  the  participants  in  each  group  ranked  these  two  characteristics  in  their  top  five  for 
“Worst  Job.” 
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Table  15.  Section  B:  Describing  Your  Ideal  and  Worst  Jobs  -  Most  Frequently  Endorsed  “ Most  Ideal  Job  ”  and  “ Worst  Job  ”  Characteristics  by 
Gender  for  the  Enlisted  Basic  Trainee  Sample 


Frequency 

Percent  of  Total 

Item 

Characteristic 

1 

2 

3 

4 

5 

Total 

1 

2 

3 

4 

5 

m 

Males 

u 
l — 

17 

Opportunities  to  work  with  cutting  edge  aircraft  technology. 

7 

5 

6 

4 

3 

25 

28.0 

20.0 

24.0 

16.0 

12.0 

l S) 

cd 

13 

Taking  lethal  action  against  enemy  targets  when  authorized  to  do  so. 

9 

3 

6 

2 

3 

23 

39.1 

13.0 

26.1 

8.7 

13.0 

1 — 
u 

2 

Opportunities  to  protect  or  rescue  friendly  forces  from  harm. 

10 

3 

4 

2 

1 

20 

50.0 

15.0 

20.0 

10.0 

5.0 

< 

Cd 

< 

14 

Participating  in  combat  operations  from  a  remote  location. 

2 

4 

4 

4 

5 

19 

10.5 

21.1 

21.1 

21.1 

26.3 

X 

u 

26 

Opportunity  to  work  in  an  emerging  career  field. 

2 

6 

5 

2 

4 

19 

10.5 

31.6 

26.3 

10.5 

21.1 

CQ 

o 

Females 

_ i 

< 

10 

Working  as  part  of  a  team  (instead  of  alone)  to  get  tasks  completed. 

9 

3 

6 

3 

2 

23 

39.1 

13.0 

26.1 

13.0 

8.7 

LU 

Q 

2 

Opportunities  to  protect  or  rescue  friendly  forces  from  harm. 

3 

2 

6 

4 

3 

18 

16.7 

11.1 

33.3 

22.2 

16.7 

1 — 

LT) 

26 

Opportunity  to  work  in  an  emerging  career  field. 

4 

8 

0 

4 

1 

17 

23.5 

47.1 

0.0 

23.5 

5.9 

O 

17 

Opportunities  to  work  with  cutting  edge  aircraft  technology. 

3 

2 

3 

2 

4 

14 

21.4 

14.3 

21.4 

14.3 

28.6 

1 

Making  decisions  that  significantly  impact  the  well-being  of  others. 

7 

4 

1 

1 

0 

13 

53.8 

30.8 

7.7 

7.7 

0.0 

Males 

32 

Potentially  limiting  the  amount  of  time  you  spend  with  family  or  friends. 

19 

4 

1 

4 

2 

30 

63.3 

13.3 

3.3 

13.3 

6.7 

22 

Spending  most  of  the  work  shift  in  a  windowless  room,  viewing  multiple  computer 
monitors. 

15 

6 

3 

2 

3 

29 

51.7 

20.7 

10.3 

6.9 

10.3 

IT 

3 

Providing  negative  performance  feedback  to  co-workers. 

3 

3 

9 

5 

5 

25 

12.0 

12.0 

36.0 

20.0 

20.0 

_  u 

CQ  1— 

7 

Frequent  observation  and  monitoring  of  your  actions  by  others. 

3 

5 

3 

6 

2 

19 

15.8 

26.3 

15.8 

31.6 

10.5 

QJ  _ 

— >  Cd 

30 

Few  opportunities  to  get  to  know  your  co-workers. 

1 

5 

4 

5 

3 

18 

5.6 

27.8 

22.2 

27.8 

16.7 

U  i— 
cd  u 

O  < 

>  cc 
>  < 

-  X 

Females 

32  Potentially  limiting  the  amount  of  time  you  spend  with  family  or  friends. 

11 

5 

5 

3 

1 

25 

44.0 

20.0 

20.0 

12.0 

4.0 

u 

22 

Spending  most  of  the  work  shift  in  a  windowless  room,  viewing  multiple  computer 
monitors. 

7 

7 

2 

1 

3 

20 

35.0 

35.0 

10.0 

5.0 

15.0 

24 

Performing  actions  that  could  seriously  injure  or  kill  others  (non-friendly  and  friendly). 

4 

3 

3 

6 

3 

19 

21.1 

15.8 

15.8 

31.6 

15.8 

30 

Few  opportunities  to  get  to  know  your  co-workers. 

5 

4 

1 

4 

4 

18 

27.8 

22.2 

5.6 

22.2 

22.2 

3 

Providing  negative  performance  feedback  to  co-workers. 

4 

2 

2 

3 

3 

14 

28.6 

14.3 

14.3 

21.4 

21.4 
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Table  16.  Section  B:  Describing  Your  Ideal  and  Worst  Jobs  -  Most  Frequently  Endorsed  “ Most  Ideal  Job”  and  “ Worst  Job”  Characteristics  by 
Gender  for  the  Officer  Trainee  Sample 


Frequency 

Percent  of  Total 

Item 

Characteristic 

1 

2 

3 

4 

5 

Total 

1 

2 

3 

4 

5 

Males 

IS) 

U 

17 

Opportunities  to  work  with  cutting  edge  aircraft  technology. 

23 

13 

9 

7 

5 

57 

40.4 

22.8 

15.8 

12.3 

8.8 

1 — 

LT) 

26 

Opportunity  to  work  in  an  emerging  career  field. 

5 

11 

10 

11 

6 

43 

11.6 

25.6 

23.3 

25.6 

14.0 

LU 

1 — 

10 

Working  as  part  of  a  team  (instead  of  alone)  to  get  tasks  completed. 

9 

12 

7 

3 

5 

36 

25.0 

33.3 

19.4 

8.3 

13.9 

u 

< 

36 

Ability  to  see  how  your  work  contributes  to  the  broader  mission,  even  if  its  importance  is  not 

5 

7 

7 

5 

8 

32 

15.6 

21.9 

21.9 

15.6 

25.0 

C£ 

< 

always  clear  and  direct. 

X 

u 

12 

Handling  mentally  challenging  situations  and  issues. 

8 

5 

3 

7 

4 

27 

29.6 

18.5 

11.1 

25.9 

14.8 

CO 

o 

Females 

36 

Ability  to  see  how  your  work  contributes  to  the  broader  mission,  even  if  its  importance  is  not 

1 

4 

0 

2 

1 

8 

12.5 

50.0 

0.0 

25.0 

12.5 

< 

LU 

always  clear  and  direct. 

g 

10 

Working  as  part  of  a  team  (instead  of  alone)  to  get  tasks  completed. 

3 

1 

1 

1 

1 

7 

42.9 

14.3 

14.3 

14.3 

14.3 

1 — 

LSI 

o 

1 

Making  decisions  that  significantly  impact  the  well-being  of  others. 

2 

1 

1 

0 

2 

6 

33.3 

16.7 

16.7 

0.0 

33.3 

p 

2 

Opportunities  to  protect  or  rescue  friendly  forces  from  harm. 

2 

2 

0 

1 

1 

6 

33.3 

33.3 

0.0 

16.7 

16.7 

26 

Opportunity  to  work  in  an  emerging  career  field. 

1 

1 

2 

1 

0 

5 

20.0 

20.0 

40.0 

20.0 

0.0 

Males 

32 

Potentially  limiting  the  amount  of  time  you  spend  with  family  or  friends. 

24 

7 

7 

7 

6 

51 

47.1 

13.7 

13.7 

13.7 

11.8 

22 

Spending  most  of  the  work  shift  in  a  windowless  room,  viewing  multiple  computer 
monitors. 

19 

7 

5 

6 

4 

41 

46.3 

17.1 

12.2 

14.6 

9.8 

LSI 

30 

Few  opportunities  to  get  to  know  your  co-workers. 

2 

10 

12 

11 

2 

37 

5.4 

27.0 

32.4 

29.7 

5.4 

_  u 

CO  1 — 

7 

Frequent  observation  and  monitoring  of  your  actions  by  others. 

6 

11 

4 

5 

2 

28 

21.4 

39.3 

14.3 

17.9 

7.1 

O  — 

— 1  cc 

31 

Limited  interactions  with  co-workers  outside  of  work. 

2 

5 

4 

4 

8 

23 

8.7 

21.7 

17.4 

17.4 

34.8 

l/l  1— 

cc  u 
o  < 

Females 

5  < 

22 

Spending  most  of  the  work  shift  in  a  windowless  room,  viewing  multiple  computer 

0 

1 

5 

1 

1 

8 

0.0 

12.5 

62.5 

12.5 

12.5 

X 

u 

32 

monitors. 

Potentially  limiting  the  amount  of  time  you  spend  with  family  or  friends. 

6 

0 

1 

1 

0 

8 

75.0 

0.0 

12.5 

12.5 

0.0 

30 

Few  opportunities  to  get  to  know  your  co-workers. 

0 

2 

0 

2 

2 

6 

0.0 

33.3 

0.0 

33.3 

33.3 

7 

Frequent  observation  and  monitoring  of  your  actions  by  others. 

0 

3 

0 

1 

0 

4 

0.0 

75.0 

0.0 

25.0 

0.0 

9 

Remaining  alert  and  highly- focused  for  long  periods  of  time,  even  when  little  is  happening. 

1 

1 

2 

0 

0 

4 

25.0 

25.0 

50.0 

0.0 

0.0 
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Table  1 7.  Section  B:  Describing  Your  Ideal  and  Worst  Jobs  -  Comparison  of  the  Most  Frequently  Endorsed  “ Most  Ideal  Job ”  and  “ Worst  Job” 
Characteristics  between  the  Total  Enlisted  Basic  Trainee  and  Officer  Trainee  Samples 


Frequency 

Percent  of  Total 

Item 

Characteristic 

1 

2 

3 

4 

5 

Total 

1 

2 

3 

4 

5 

LO 

U 

Enlisted  Basic  Trainees 

10 

Working  as  part  of  a  team  (instead  of  alone)  to  get  tasks  completed. 

10 

11 

8 

6 

4 

39 

25.6 

28.2 

20.5 

15.4 

10.3 

LO 

CH 

17 

Opportunities  to  work  with  cutting  edge  aircraft  technology. 

10 

7 

9 

6 

7 

39 

25.6 

17.9 

23.1 

15.4 

17.9 

LU 

1 — 
u 

2 

Opportunities  to  protect  or  rescue  friendly  forces  from  harm. 

13 

5 

10 

6 

4 

38 

34.2 

13.2 

26.3 

15.8 

10.5 

< 

cc 

26 

Opportunity  to  work  in  an  emerging  career  field. 

6 

14 

5 

6 

5 

36 

16.7 

38.9 

13.9 

16.7 

13.9 

< 

I 

u 

13 

Taking  lethal  action  against  enemy  targets  when  authorized  to  do  so. 

9 

5 

6 

4 

3 

27 

33.3 

18.5 

22.2 

14.8 

11.1 

CD 

Officer  Trainees 

_ l 

17 

Opportunities  to  work  with  cutting  edge  aircraft  technology. 

25 

14 

10 

7 

5 

61 

41.0 

23.0 

16.4 

11.5 

8.2 

< 

LU 

o 

26 

Opportunity  to  work  in  an  emerging  career  field. 

7 

12 

12 

12 

6 

49 

14.3 

24.5 

24.5 

24.5 

12.2 

1 — 

10 

Working  as  part  of  a  team  (instead  of  alone)  to  get  tasks  completed. 

12 

14 

9 

4 

6 

45 

26.7 

31.1 

20.0 

8.9 

13.3 

o 

36 

Ability  to  see  how  your  work  contributes  to  the  broader  mission,  even  if  its  importance  is  not 

6 

11 

8 

7 

9 

41 

14.6 

26.8 

19.5 

17.1 

22.0 

p 

2 

always  clear  and  direct. 

Opportunities  to  protect  or  rescue  friendly  forces  from  harm. 

11 

7 

7 

2 

7 

34 

32.4 

20.6 

20.6 

5.9 

20.6 

Enlisted  Basic  Trainees 

32 

Potentially  limiting  the  amount  of  time  you  spend  with  family  or  friends. 

30 

9 

6 

7 

3 

55 

54.5 

16.4 

10.9 

12.7 

5.5 

22 

Spending  most  of  the  work  shift  in  a  windowless  room,  viewing  multiple  computer 
monitors. 

22 

13 

5 

3 

6 

49 

44.9 

26.5 

10.2 

6.1 

12.2 

ir 

3 

Providing  negative  performance  feedback  to  co-workers. 

7 

5 

11 

8 

8 

39 

17.9 

12.8 

28.2 

20.5 

20.5 

_  y 

CO  l— 

30 

Few  opportunities  to  get  to  know  your  co-workers. 

6 

9 

5 

9 

7 

36 

16.7 

25.0 

13.9 

25.0 

19.4 

— 1  cc 

1—  LU 

LO  |— 

7 

Frequent  observation  and  monitoring  of  your  actions  by  others. 

3 

6 

5 

9 

5 

28 

10.7 

21.4 

17.9 

32.1 

17.9 

CC  u 

o  < 

Officer  Trainees 

< 

"  X 

32 

Potentially  limiting  the  amount  of  time  you  spend  with  family  or  friends. 

31 

8 

8 

8 

6 

61 

50.8 

13.1 

13.1 

13.1 

9.8 

u 

22 

Spending  most  of  the  work  shift  in  a  windowless  room,  viewing  multiple  computer 
monitors. 

20 

8 

10 

8 

5 

51 

39.2 

15.7 

19.6 

15.7 

9.8 

30 

Few  opportunities  to  get  to  know  your  co-workers. 

2 

12 

12 

13 

5 

44 

4.5 

27.3 

27.3 

29.5 

11.4 

7 

Frequent  observation  and  monitoring  of  your  actions  by  others. 

6 

14 

5 

6 

2 

33 

18.2 

42.4 

15.2 

18.2 

6.1 

9 

Remaining  alert  and  highly-focused  for  long  periods  of  time,  even  when  little  is  happening. 

3 

4 

8 

6 

4 

25 

12.0 

16.0 

32.0 

24.0 

16.0 
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SECTION  C:  YOUR  REACTIONS  TO  THE  INVENTORY 


The  top  half  of  Table  18  summarizes  enlisted  trainees’  reactions  to  the  inventory  by 
gender.  Overall,  male  enlisted  trainees  (M=  4.32,  SD  =  .59)  reacted  significantly  more  positively 
to  the  inventory,  on  average,  than  female  enlisted  trainees  (M=  3.99,  SD  =  .59)  (d  =  .56,  p  <  .05). 
Specifically,  male  trainees  rated  the  ease  of  the  ranking  exercise  and  the  clarity  of  the 
instructions  significantly  higher  than  did  female  trainees  ( d  =  .49,  p  <  .05;  d  =  .53, p  <.05. 

The  bottom-half  of  Table  18  summarizes  officer  trainees’  reactions  to  the  inventory  by 
gender.  Overall,  female  officer  trainees  (M=  4.28,  SD  =  .40)  rated  the  inventory  significantly 
more  positively  than  male  officer  trainees  (M=  3.92,  SD  =  .55)  ( d  =  -.68,  p  <  .05).  In 
particular,  female  officer  trainees  rated  the  ease  of  the  ranking  exercise  significantly  higher 
than  male  trainees  (d=  -.91, p  <  .05). 


Table  18.  Section  C:  Your  Reactions  to  the  Inventory  -  Comparison  between  Male  and 
Female  Trainees  by  Sample 


Male 

Female 

Male-Female 

Sample/Item 

Mean 

SD 

n 

Mean 

SD 

n 

Diff 

d 

t 

P 

Enlisted  Basic  Trainees  (n  =  89) 

1 .  Ease  of  Ranking 

4.15 

.95 

48 

3.68 

.99 

41 

.46 

.49 

2.26 

.03 

Exercise  (Section  B) 

3a.  Instructions  Clarity 

4.48 

.65 

48 

4.12 

.71 

41 

.36 

.53 

2.47 

.02 

3b.  Item  Clarity 

4.33 

.75 

48 

4.17 

.67 

41 

.16 

.23 

1.07 

.29 

Overall 

4.32 

.59 

48 

3.99 

.59 

41 

.33 

.56 

2.62 

.01 

Officer  Trainees  (n  =  96) 

1.  Ease  of  Ranking 

Exercise  (Section  B) 

3.69 

.90 

83 

4.46 

.52 

13 

-.77 

-.91 

-3.03 

.00 

3a.  Instructions  Clarity 

4.10 

.64 

83 

4.23 

.73 

13 

-.13 

-.21 

-.69 

.49 

3b.  Item  Clarity 

3.99 

.76 

83 

4.15 

.55 

13 

-.17 

-.23 

-.76 

.45 

Overall 

3.92 

.55 

83 

4.28 

.40 

13 

-.36 

-.68 

-2.24 

.03 

Note.  All  questions  rated  on  a  5-point  scale,  with  anchors  for  question  1  ranging  from  "Very  Difficult"  to  "Very  Easy" 
and 

anchors  for  questions  3a  and  3b  ranging  from  "Very  Bad"  to  "Very  Good."  Diff=  raw  difference  in  mean  scores,  d 
=  standardized  mean  difference  or  Cohen’s  d.  Negative  Diff  and  d  values  mean  that  female  enlisted  basic  trainees 
endorsed  an  item  more  frequently,  on  average,  than  male  trainees.  Statically  significant  d  values  ( t  >  1.96,  p  <  .05) 
are  in  bold  typeface. 


Table  19  summarizes  reactions  to  the  inventory  for  both  the  enlisted  and  officer  trainee 
samples.  Both  samples  generally  reacted  positively  to  the  inventory.  The  mean  score  for  both 
groups  was  near  the  scale  anchor  “good.”  Overall,  enlisted  trainees  tended  to  react  more 
positively  to  the  inventory  than  officer  trainees.  However,  the  differences  between  the  two 
samples  were  not  statistically  significant  (p  >.05)  or  were  generally  small  in  magnitude:  ease  of 
ranking  exercise  (d  =  .  1 1),  clarity  of  instructions  (d  =  .34),  clarity  of  items  (d  =  .33),  and  overall 
(< d  =  .33).  The  majority  of  both  samples  provided  a  mean  rating,  across  reaction  questions,  of  4.0 
or  higher  (70%  of  enlisted  trainees  and  60%  of  officer  trainees)  and  none  provided  a  mean  rating 
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lower  than  2.0.  Both  samples  reacted  more  positively  to  the  clarity  of  the  instructions  and  to  the 
items  than  to  the  ranking  exercise. 


Table  19.  Section  C:  Your  Reactions  to  the  Inventory  -  Comparison  between  Total  Enlisted 
Basic  and  Officer  Trainee  Samples  in  Item-  and  Scale-Level  Descriptive  Statistics 


Samnle/Item 

Mean 

Median 

SD 

Min 

Max 

Skew 

Kurt 

Valid 

Resv 

Missing 

Resn 

% 

Valid 

Enlisted  Basic  Trainees 

1.  Ease  of  Ranking 
Exercise  (Section  B) 

3.91 

4 

1.00 

1 

5 

-.57 

-.43 

90 

0 

100 

3a.  Instructions  Clarity 

4.32 

4 

.70 

3 

5 

-.54 

-.82 

90 

0 

100 

3b.  Item  Clarity 

4.26 

4 

.71 

3 

5 

-.42 

-.94 

90 

0 

100 

Overall 

4.16 

4 

.61 

2.67 

5.00 

-.21 

-.70 

90 

0 

100 

Officer  Trainees 

1 .  Ease  of  Ranking 
Exercise  (Section  B) 

3.81 

4 

.90 

1 

5 

-.47 

-.04 

99 

2 

98 

3a.  Instructions  Clarity 

4.09 

4 

.66 

3 

5 

-.10 

-.64 

99 

2 

98 

3b.  Item  Clarity 

4.02 

4 

.73 

2 

5 

-.52 

.36 

99 

2 

98 

Overall 

3.97 

4 

.54 

2.67 

5.00 

-.20 

-.83 

99 

2 

98 

Note.  All  questions  rated  on  a  5-point  scale,  with  anchors  for  question  1  ranging  from  very  difficult  to  very  easy 
and  anchors  for  questions  3a  and  3b  ranging  from  very  bad  to  very  good.  Valid  Resp  =  Number  of  valid 
responses,  Missing  Resp  =  Number  of  missing  responses,  %  Valid  =  Percentage  of  valid  responses.  Items  rated 
on  a  scale  from  1  (“Very  Difficult”  for  item  1,  “Very  Bad”  for  items  3a  and  3b)  to  5  (“Very  Easy”  for  item  1, 
“Very  Good”  for  items  3a  and  3b). 
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Project  Summary 


We  began  by  integrating  information  about  the  SAOCs  required  in  the  RPA  pilot  and  SO 
jobs.  We  used  the  0*NET  content  model  as  an  organizing  structure  to  ensure  that  we  covered  all 
of  the  important  domains  (e.g.,  abilities,  skills,  work  style  preferences,  work  context)  and  to 
provide  a  well-researched  taxonomic  structure  and  labels  for  the  disparate  sources  of  information. 
RPA  SMEs  and  the  AFPC  project  team  helped  us  narrow  the  list  to  those  SAOCs  that  would  be 
most  critical  to  measure  in  an  entry-level  selection  process.  Next,  we  worked  with  the  AFPC 
team  to  take  account  of  practical  considerations,  such  as  realistic  constraints  on  testing  time  and 
how  the  predictor  measures  could  fit  into  existing  enlisted  and  officer  selection  processes. 
Ultimately,  we  provided  two  options  for  recommended  entry-level  selection  batteries.  One  option 
makes  maximal  use  of  existing  operational  selection  tools  and  administration  processes;  the 
second  replaces  some  of  the  existing  selection  tools  with  others  that  have  demonstrated  evidence 
of  reliability  and  validity,  as  well  as  at  least  some  evidence  that  they 
result  in  smaller  gender  subgroup  score  differences  (and  possibly  smaller  race/ethnic 
subgroup  score  differences). 

We  were  also,  within  the  scope  of  this  project,  to  develop  two  new  predictor  measures  to 
address  important  measurement  gaps  without  adding  substantially  to  the  testing  burden  for 
enlisted  or  officer  candidates.  The  first  is  a  measure  of  time  sharing  ability  that  involves 
perfonning  multiple  tasks  tapping  working  memory,  task  prioritization,  and  selective  attention. 
This  measure  can  be  administered  on  the  existing  TBAS  platform  and  it  has  a  modular 
programming  foundation  that  provides  a  great  deal  of  flexibility  for  future  research  and 
adaptations.  This  measure  could  be  used  for  selection  into  any  career  field  that  requires  time 
sharing  ability,  and  ican  also  be  used  for  basic  research  on  time  sharing  abilities.  The  second 
new  measure  is  an  RPA-specific  P-E  fit  measure.  This  instrument  is  best  suited  for  use  as  a  self- 
assessment  tool  that  can  help  potential  recruits  determine  if  the  RPA  work  context  would  be  a 
good  fit  for  their  work  preferences.  It  could  open  a  door  for  recruiters  to  discuss  the  RPA  career 
field  prior  to  enlistment  or  accessioning. 

We  believe  that  adding  the  MT  Test  to  the  selection  process  will  provide  incremental 
validity  beyond  existing  measures,  but  this  is,  of  course,  an  empirical  question.  We  also  believe 
that  the  P-E  fit  measure  can  assist  with  recruiting  candidates  into  the  RPA  career  field. 

In  the  future,  the  USAF  could  consider  supplementing  the  recommended  predictor 
batteries  with  part  or  all  of  the  Predator  selection  battery  being  developed  by  the  UK  RAF.  The 
USAF  could  also  consider  developing  measures  to  address  other  measurement  gaps  identified  in 
this  project,  including  judgment  and  decision  making,  critical  thinking  skills,  teamwork  skills, 
and/or  oral  expression  and  comprehension  skills.  Prior  to  developing  such  measures,  we 
recommend  further  discussion  and  exploration  of  the  likely  increment  in  validity  beyond  existing 
measures,  relative  to  the  likely  cost  of  increasing  test  administration  time  and/or  adding  a 
requirement  for  non-standard  equipment  (e.g.,  the  capability  to  capture  audio  input). 
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Table  A.l.  Skills  relevant  to  RPA  Pilot  and  SO  occupations6. 


Skill  Type 

Skill 

Definition 

BASIC  SKILLS 
-  Developed 
capacities  that 
facilitate 
learning  or  the 
more  rapid 

Content  Skills  -  Basic 

skills  needed  to  work  with 
and  acquire  more  specific 
skills  in  a  variety  of 
domains. 

Active  Listening 

Giving  full  attention  to  what  other  people  are  saying,  taking  time  to 
understand  the  points  being  made,  asking  questions  as  appropriate,  and  not 
interrupting  at  inappropriate  times 

acquisition  of 
knowledge 

Process  Skills  - 

Procedures  that 

contribute  to  the  more 
rapid  acquisition  of 
knowledge  and  skill 
across  a  variety  of 
domains 

Critical  Thinking 

Using  logic  and  reasoning  to  identify  the  strengths  and  weaknesses  of 
alternative  solutions,  conclusions  or  approaches  to  problems. 

CROSS 

FUNCTIONAL 

SKILLS  - 

Social  Skills  -  Developed 
capacities  used  to  work 
with  people  to  achieve 

Social 

Perceptiveness 

Being  aware  of  others'  reactions  and  understanding  why  they  react  as  they 
do. 

Developed 
capacities  that 
facilitate 
performance  of 
activities  that 
occur  across  jobs 

goals 

Coordination 

Adjusting  actions  in  relation  to  others'  actions 

Persuasion 

Persuading  others  to  change  their  minds  or  behavior 

Instructing 

Teaching  others  how  to  do  something 

6  Skill  categories,  titles,  and  definitions  are  from  the  0*NET  taxonomy.  (See  http://www.onetcenter.org/content.html) 

Page  75 


Skill  Type 

Skill 

Definition 

Complex  Problem  Solving 
-  Identifying  complex 
problems  and  reviewing 
related  information  to 
develop  and  evaluate 
options  and  implement 
solutions 

Complex 

Problem  Solving 

Identifying  complex  problems  and  reviewing  related  information  to  develop 
and  evaluate  options  and  implement  solutions 

Systems  Skills  -  Developed 
capacities  used  to 
understand,  monitor,  and 

Systems  Analysis 

Determining  how  a  system  should  work  and  how  changes  in  conditions, 
operations,  and  the  environment  will  affect  outcomes 

improve  socio-technical 
systems 

Judg.  and  Dec. 
Making 

Considering  the  relative  costs  and  benefits  of  potential  actions  to  choose  the 
most  appropriate  one 

Resource  Management 
Skills  -  Developed 
capacities  used  to  allocate 
resources  efficiently 

Time 

Management 

Managing  one's  own  time  and  the  time  of  others 
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Table  A.2.  Abilities  relevant  to  RPA  Pilot  and  SO  occupations. 7 


Ability  Category 

Ability 

Definition 

Verbal  Abilities  - 

Abilities  that  influence 
the  acquisition  and 

Oral  Comprehension 

The  ability  to  listen  to  and  understand  information  and  ideas  presented  through 
spoken  words  and  sentences. 

application  of  verbal 
information  in  problem 

Written  Comprehension 

The  ability  to  read  and  understand  information  and  ideas  presented  in  writing. 

solving 

Oral  Expression 

The  ability  to  communicate  information  and  ideas  in  speaking  so  others  will 
understand. 

Problem  Sensitivity 

The  ability  to  tell  when  something  is  wrong  or  is  likely  to  go  wrong.  It  does  not 
involve  solving  the  problem,  only  recognizing  there  is  a  problem 

Deductive  Reasoning 

The  ability  to  apply  general  rules  to  specific  problems  to  produce  answers  that  make 
sense. 

Inductive  Reasoning 

The  ability  to  combine  pieces  of  information  to  form  general  rules  or  conclusions 
(includes  finding  a  relationship  among  seemingly  unrelated  events). 

Information  Ordering 

The  ability  to  arrange  things  or  actions  in  a  certain  order  or  pattern  according  to  a 
specific  rule  or  set  of  rules  (e.g.,  patterns  of  numbers,  letters,  words,  pictures, 
mathematical  operations). 

Quantitative  Abilities- 

Abilities  that  influence 
the  solution  of  problems 

Mathematical  Reasoning 

The  ability  to  choose  the  right  mathematical  methods  or  formulas  to  solve  a 
problem. 

involving  mathematical 
relationships 

Number  Facility 

The  ability  to  add,  subtract,  multiply,  or  divide  quickly  and  correctly. 

1  With  the  exception  of  “Situational  Awareness,”  ability  categories,  titles,  and  definitions  are  taken  from  the  0*NET  taxonomy.  (See 
http://www.onetcenter.org/content.html) 
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Ability  Category 

Ability 

Definition 

Memory-  Abilities 
related  to  the  recall  of 
available  information 

Working  Memory** 

The  ability  to  temporarily  hold  information  in  memory  while  processing  other 
information. 

Memorization 

The  ability  to  remember  information  such  as  words,  numbers,  pictures,  and  procedures. 

Flexibility  of  Closure 

The  ability  to  identify  or  detect  a  known  pattern  (a  figure,  object,  word,  or  sound) 
that  is  hidden  in  other  distracting  material. 

Perceptual  Speed 

The  ability  to  quickly  and  accurately  compare  similarities  and  differences  among 
sets  of  letters,  numbers,  objects,  pictures,  or  patterns.  The  things  to  be  compared 
may  be  presented  at  the  same  time  or  one  after  the  other. 

Spatial  Abilities  - 

Abilities  related  to  the 
manipulation  and 

Spatial  Orientation 

The  ability  to  know  your  location  in  relation  to  the  environment  or  to  know  where 
other  objects  are  in  relation  to  you. 

organization  of  spatial 
information 

Visualization 

The  ability  to  imagine  how  something  will  look  after  it  is  moved  around  or  when  its 
parts  are  moved  or  rearranged. 

Attentiveness  -  Abilities 
related  to  application  of 

Selective  Attention 

The  ability  to  concentrate  on  a  task  over  a  period  of  time  without  being  distracted. 

attention 

Time  Sharing 

The  ability  to  shift  back  and  forth  between  two  or  more  activities  or  sources  of 
information  (such  as  speech,  sounds,  touch,  or  other  sources). 

Fine  Manipulative 
Abilities  -  Abilities 
related  to  the 

Arm-FIand  Steadiness 

The  ability  to  keep  your  hand  and  arm  steady  while  moving  your  arm  or  while 
holding  your  arm  and  hand  in  one  position. 

manipulation  of  objects 

Manual  Dexterity 

The  ability  to  quickly  move  your  hand,  your  hand  together  with  your  arm,  or  your 
two  hands  to  grasp,  manipulate,  or  assemble  objects. 

Finger  Dexterity 

The  ability  to  quickly  move  your  hand,  your  hand  together  with  your  arm,  or  your 
two  hands  to  grasp,  manipulate,  or  assemble  objects. 
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Ability  Category 

Ability 

Definition 

Control  Movement 
Abilities  -  Abilities 
related  to  the  control 

Control  Precision 

The  ability  to  quickly  and  repeatedly  adjust  the  controls  of  a  machine  or  a  vehicle  to 
exact  positions. 

and  manipulation  of 
objects  in  time  and  space 

Multi-limb  Coordination 

The  ability  to  coordinate  two  or  more  limbs  (for  example,  two  arms,  two  legs,  or  one 
leg  and  one  arm)  while  sitting,  standing,  or  lying  down.  It  does  not  involve 
performing  the  activities  while  the  whole  body  is  in  motion. 

Response  Orientation 

The  ability  to  choose  quickly  between  two  or  more  movements  in  response  to  two  or 
more  different  signals  (lights,  sounds,  pictures).  It  includes  the  speed  with  which  the 
correct  response  is  started  with  the  hand,  foot,  or  other  body  part. 

Rate  Control 

The  ability  to  time  your  movements  or  the  movement  of  a  piece  of  equipment  in 
anticipation  of  changes  in  the  speed  and/or  direction  of  a  moving  object  or  scene. 

Reaction  Time  and 
Speed  Abilities  -Abilities 
related  to  speed  of 
manipulation  of  objects 

Reaction  Time 

The  ability  to  quickly  respond  (with  the  hand,  finger,  or  foot)  to  signal  (sound,  light, 
picture)  when  it  appears. 

Visual  Abilities  -  Abilities  related  to  visual  sensory  input  (Near, 

Far,  Night,  &  Peripheral  Vision,  Visual  Color  Discrimination,  Depth 
Perception  and  Glare  Sensitivity) 


Auditory  and  Speech  Abilities  -  Abilities  related  to  auditory  and 
oral  input 


Physical  Strength  -  Abilities  related  to  the  capacity  to  exert  force 
(Static,  Explosive,  Dynamic,  &  Trunk  strength) 


Endurance  -  The  ability  to  exert  oneself  physically  over  long 
periods  without  getting  out  of  breath  (Stamina,  Extent  Flexibility, 
Dynamic  Flexibility,  &  Gross  Body  Coordination  and/or 
Equilibrium) 
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Definition 


Ability  Category  Ability 


Situational  Awareness  (Non-0*NET)  The  ability  to  maintain  the  wide  breadth  of  awareness  and  a  fully  cognizant  state. 

Situational  Awareness  represents  the  combined  effectiveness  of  the  perceptual 
system,  short  term  memory,  reasoning  ability,  spatial  processing,  psychomotor 
abilities  and  information  processing  (i.e.,  the  effective  capacity  of  the  whole  cognitive 
system). 
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Table  A3.  Work  Styles  relevant  to  RPA  Pilot  and  SO  occupations .s 


Category 

Work  Style 

Definition 

Achievement  Orientation  -  Setting 
personal  goals,  trying  to  succeed  at 
those  goals,  and  striving  to  be 
competent  in  own  work 

Achievement/Effort 

Establishing  and  maintaining  personally  challenging  achievement 
goals  and  exerting  effort  toward  mastering  tasks. 

Persistence 

Persisting  in  the  face  of  obstacles. 

Initiative 

Willingness  to  take  on  responsibilities  and  challenges. 

Social  Influence  -  Having  an  impact  on 
others  in  the  organization,  and 
displaying  energy >  and  leadership 

Leadership 

Willingness  to  lead,  take  charge,  and  offer  opinions  and  direction. 

Interpersonal  Orientation  -  Being 

Cooperation 

Being  pleasant  with  others  on  the  job  and  displaying  a  good-natured, 

pleasant,  cooperative,  sensitive  to  others, 
easy  to  get  along  with,  and  having  a 
preference  for  associating  with 
other  organization  members 

cooperative  attitude. 

Adjustment  -  Maturity,  poise,  flexibility, 
and  restraint  to  cope  with  pressure, 
stress,  criticism,  setbacks,  personal  and 
work-related  problems,  etc. 

Self  Control 

Maintaining  composure,  keeping  emotions  in  check,  controlling 
anger,  and  avoiding  aggressive  behavior,  even  in  very  difficult 
situations. 

Stress  Tolerance 

Accepting  criticism  and  dealing  calmly  and  effectively  with  high 
stress  situations. 

Adaptability/Flexibility 

Being  open  to  change  (positive  or  negative)  and  to  considerable 
variety  in  the  workplace. 

8  Work  Style  categories,  titles,  and  definitions  are  taken  from  the  0*NET  taxonomy.  (See  http://www.onetcenter.org/content.html) 
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Category 

Work  Style 

Definition 

Conscientiousness  -  Dependability, 
commitment  to  doing  the  job  correctly 
and  carefully,  and  being  trustworthy, 
accountable,  and  attentive  to  details 

Dependability 

Requires  being  reliable,  responsible,  and  dependable,  and  fulfilling 
obligations. 

Integrity 

Being  honest  and  ethical. 

Independence 

Developing  one's  own  ways  of  doing  things,  guiding  oneself  with 
little  or  no  supervision,  and  depending  on  oneself  to  get  things  done. 

Practical  Intelligence  -  Generating 
useful  ideas  and  thinking  things  through 
logically 

Analytical  Thinking 

Analyzing  information  and  using  logic  to  address  work-related 
issues  and  problems. 
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Appendix  B 

Information  about  Existing  Measures  that  Could  be  Used  for  RPA  Operator  Selection 


Appendix  B_Potential 
Predictors  for  RPA  O 
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Appendix  C 

Original  List  of  Work  Context  Factors 
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Source 

# 

Work  Context  Statement 

Army  ONET 

1 

Required  to  work  indoors  or  in  an  environmentally  controlled  environment. 

Army  ONET 

2 

Required  to  work  in  enclosed  or  cramped  spaces. 

Army  ONET 

3 

Exposed  to  sounds  and  noise  levels  that  are  distracting  and  uncomfortable. 

Army  ONET 

4 

Exposed  to  extreme  temperatures  (hot  or  cold). 

Army  ONET 

5 

Required  to  work  at  night  or  under  inadequate  lighting  conditions. 

Army  ONET 

6 

Exposed  to  significant  levels  of  physical  or  mental  discomfort,  stress,  or 
strain. 

Army  ONET 

7 

Required  to  work  with  limited,  inadequate,  or  defective  equipment,  supplies, 
or  materials. 

Army  ONET 

8 

Required  to  work  long  hours  or  take  on  additional  duties  due  to  limited  or 
inadequate  personnel. 

Army  ONET 

9 

Required  to  perform  their  duties  with  limited  or  inadequate  information  or 
instructions. 

Army  ONET 

10 

Required  to  work  through  frequent  or  unscheduled  interruptions  that  make  it 
hard  to  complete  their  work  on  time. 

Army  ONET 

11 

Required  to  respond  to  frequent  crises  or  emergencies. 

Army  ONET 

12 

Afforded  the  freedom  to  determine  the  timing  and  scheduling  of  their  work. 

Army  ONET 

13 

Afforded  the  freedom  to  determine  which  methods  and  procedures  are  used 
to  complete  their  work. 

Army  ONET 

14 

Required  to  work  at  a  pace  or  speed  determined  by  equipment  or  technology. 

Army  ONET 

15 

Required  to  make  decisions  that  affect,  or  coidd  affect,  others. 

Army  ONET 

16 

Required  to  perform  duties  of  long  or  extended  duration. 

Army  ONET 

17 

Required  to  work  long  or  extended  hours  with  little  to  no  sleep. 

Army  ONET 

18 

Afforded  limited  rest  or  recovery  time  between  stressful  or  demanding  tasks. 

Army  ONET 

19 

Required  to  plan  or  perform  under  significant  time  pressure. 

Army  ONET 

20 

Required  to  be  very  precise  and  highly  accurate  when  completing  their  tasks. 

Army  ONET 

21 

Required  to  perform  under  circumstances  of  conflicting  or  ambiguous 
directions,  orders,  or  priorities. 
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Source 

# 

Work  Context  Statement 

Army  ONET 

22 

Required  to  work  under  minimal  or  limited  supervision. 

Army  ONET 

23 

Required  to  perform  duties  for  which  they  received  minimal  or  limited 
training. 

Army  ONET 

24 

Required  to  perform  non-job  specific  duties  or  to  complete  tasks  outside  of 
their  job. 

Army  ONET 

25 

Required  to  depend  heavily  on  technology  or  equipment  to  complete  their 
tasks. 

Army  ONET 

26 

Required  to  perform  continuous,  repetitive  physical  or  mental  tasks. 

Army  ONET 

27 

Required  to  interact  or  deal  with  non-hostile  host  country  nationals. 

Army  ONET 

28 

Required  to  interact  or  deal  with  hostile  but  non-violent  host  country 
nationals  or  groups. 

Army  ONET 

29 

Required  to  deal  with  violent  or  physically  aggressive  host  country  nationals 
or  groups. 

Army  ONET 

30 

Required  to  interact  or  work  with  non-Air  Force  personnel. 

Army  ONET 

31 

Required  to  perform  work  that  is  highly  visible  to  others  and  for  which 
others  can  observe  what  they  are  doing. 

Army  ONET 

32 

Required  to  interact  or  work  with  challenging  and  difficult  co workers. 

Army  ONET 

33 

Required  to  work  closely  with  or  depend  heavily  on  others  to  complete  their 
own  duties. 

Army  ONET 

34 

Required  to  perform  their  work  largely  on  their  own,  with  little  assistance 
from  others. 

Army  ONET 

35 

Required  to  interact  and  work  with  others  at  a  distance  (e.g.,  through  e-mail 
or  other  forms  of  electronic  communication). 

Army  ONET 

36 

Required  to  work  as  a  member  of  a  team  consisting  of  individuals  outside  of 
the  Soldier’s  unit  or  the  Air  Force. 

Army  ONET 

37 

Required  to  coordinate  or  lead  others  for  whom  the  Soldier  has  no  direct 
authority. 

Army  ONET 

38 

Required  to  persuade  or  influence  others  for  whom  the  Soldier  has  no  direct 
authority. 

Army  ONET 

39 

Required  to  be  responsible  for  the  health  and  safety  of  non-Air  Force 
personnel. 
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Source 

# 

Work  Context  Statement 

new 

40 

Required  to  maintain  high  levels  of  attention,  even  when  there  is  little 
activity. 

new 

41 

May  be  required  to  take  actions  against  persons  with  whom  they  have 
formed  a  psychological  attachment  with. 

new 

42 

Required  to  sit  for  prolonged  periods  of  time. 

new 

43 

Required  to  work  rotating  duty  shifts. 

new 

44 

May  experience  motion  sickness. 

new 

45 

Exposed  to  personal  danger  or  individual  risks,  man-made  or  natural. 

new 

46 

May  be  required  to  deploy  for  extended  periods  of  time 

new 

47 

May  be  required  to  shift  roles  between  soldier  and  civilian  immediately  and 
on  constant  basis  (i.e.,  daily). 
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Appendix  D. 

Field  Test  Version  of  the  P-E  Fit  Measure 


Note.  This  measure  has  been  removed  prior  to  publication  and  distribution. 
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